Jul 13 19:25:08.319716 ip-10-0-135-83 systemd[1]: Starting Kubernetes Kubelet... Jul 13 19:25:08.733568 ip-10-0-135-83 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 19:25:08.733568 ip-10-0-135-83 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 19:25:08.733568 ip-10-0-135-83 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 19:25:08.733568 ip-10-0-135-83 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 19:25:08.733568 ip-10-0-135-83 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 19:25:08.736397 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.736256 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 19:25:08.743756 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743737 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.743756 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743755 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743759 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743764 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743766 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743771 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743776 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743779 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743782 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743785 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743788 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743790 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743793 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743795 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743798 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743801 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743805 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743807 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743817 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743820 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.743815 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743823 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743826 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743829 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743831 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743834 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743836 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743839 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743842 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743844 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743846 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743849 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743851 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743853 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743856 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743858 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743862 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743864 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743866 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743869 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743871 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743873 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.744261 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743876 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743883 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743886 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743888 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743891 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743894 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743896 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743899 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743901 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743904 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743907 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743910 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743912 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743915 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743918 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743920 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743923 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743925 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743927 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743930 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.744782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743933 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743935 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743937 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743940 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743942 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743945 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743947 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743950 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743952 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743954 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743957 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743960 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743962 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743964 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743967 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743969 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743972 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743975 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743977 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743981 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.745273 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743984 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743986 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743988 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743991 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743993 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743996 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.743998 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744001 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744003 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744006 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744008 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744011 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744015 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744018 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744021 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744023 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744026 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.744029 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745568 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.745755 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745576 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745579 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745583 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745586 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745589 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745592 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745595 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745597 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745600 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745602 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745605 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745607 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745610 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745613 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745615 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745618 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745620 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745623 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745626 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745628 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.746199 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745631 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745633 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745636 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745639 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745641 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745643 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745646 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745648 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745650 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745653 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745655 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745658 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745660 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745663 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745665 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745667 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745669 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745672 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745675 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745677 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.746698 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745680 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745682 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745685 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745687 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745690 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745692 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745694 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745697 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745700 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745703 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745705 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745708 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745710 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745713 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745716 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745720 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745724 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745726 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745729 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745732 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.747170 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745735 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745738 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745740 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745743 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745745 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745748 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745750 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745753 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745755 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745758 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745761 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745763 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745766 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745769 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745773 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745775 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745778 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745780 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745784 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745788 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.747659 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745791 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745793 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745797 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745799 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745802 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745804 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745807 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745809 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745812 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745814 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745817 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745819 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745821 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745824 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745826 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745829 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745832 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.745835 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746201 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.748134 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746206 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746209 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746212 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746215 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746218 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746221 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746224 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746227 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746229 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746232 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746235 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746237 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746240 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746242 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746246 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746249 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746252 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746255 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746258 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.748601 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746261 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746264 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746267 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746269 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746272 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746275 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746277 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746280 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746284 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746287 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746289 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746292 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746295 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746298 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746300 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746303 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746305 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746312 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746316 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746320 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.749083 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746323 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746325 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746328 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746331 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746333 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746336 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746338 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746341 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746343 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746346 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746348 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746351 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746353 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746356 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746358 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746361 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746363 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746365 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746368 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746370 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.749591 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746373 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746375 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746377 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746379 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746382 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746384 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746387 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746389 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746392 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746394 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746402 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746405 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746407 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746410 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746412 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746415 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746417 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746419 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746438 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746441 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.750058 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746443 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746446 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746448 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746451 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746453 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746455 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746458 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746460 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746463 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746465 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746467 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746470 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746472 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746475 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746477 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746479 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746481 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746484 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.746486 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746565 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746575 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 19:25:08.750588 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746592 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746598 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746608 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746611 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746615 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746619 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746622 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746625 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746628 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746631 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746634 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746637 2582 flags.go:64] FLAG: --cgroup-root="" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746640 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746643 2582 flags.go:64] FLAG: --client-ca-file="" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746645 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746648 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746654 2582 flags.go:64] FLAG: --cluster-domain="" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746657 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746660 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746663 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746666 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746670 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746673 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 19:25:08.751077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746676 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746680 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746683 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746686 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746688 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746691 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746694 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746698 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746703 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746706 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746709 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746712 2582 flags.go:64] FLAG: --enable-server="true" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746716 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746721 2582 flags.go:64] FLAG: --event-burst="100" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746724 2582 flags.go:64] FLAG: --event-qps="50" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746728 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746731 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746734 2582 flags.go:64] FLAG: --eviction-hard="" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746738 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746741 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746744 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746747 2582 flags.go:64] FLAG: --eviction-soft="" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746750 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746753 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746755 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 19:25:08.751770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746758 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746761 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746764 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746766 2582 flags.go:64] FLAG: --feature-gates="" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746771 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746773 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746776 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746779 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746782 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746785 2582 flags.go:64] FLAG: --help="false" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746788 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-83.ec2.internal" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746791 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746794 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746796 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746800 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746805 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746808 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746811 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746813 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746816 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746820 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746823 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746827 2582 flags.go:64] FLAG: --kube-reserved="" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746829 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 19:25:08.752386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746832 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746835 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746838 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746840 2582 flags.go:64] FLAG: --lock-file="" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746843 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746846 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746849 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746854 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746857 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746860 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746862 2582 flags.go:64] FLAG: --logging-format="text" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746865 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746868 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746871 2582 flags.go:64] FLAG: --manifest-url="" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746874 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746878 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746880 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746884 2582 flags.go:64] FLAG: --max-pods="110" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746887 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746890 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746892 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746895 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746898 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746903 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746905 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 19:25:08.752995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746912 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746915 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746918 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746921 2582 flags.go:64] FLAG: --pod-cidr="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746925 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746931 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746934 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746937 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746940 2582 flags.go:64] FLAG: --port="10250" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746944 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746946 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-077eb7d141f80e844" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746949 2582 flags.go:64] FLAG: --qos-reserved="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746952 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746955 2582 flags.go:64] FLAG: --register-node="true" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746958 2582 flags.go:64] FLAG: --register-with-taints="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746961 2582 flags.go:64] FLAG: --registry-burst="10" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746964 2582 flags.go:64] FLAG: --registry-qps="5" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746967 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746969 2582 flags.go:64] FLAG: --reserved-memory="" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746972 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746975 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746978 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746981 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746983 2582 flags.go:64] FLAG: --runonce="false" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746986 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 19:25:08.753630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746989 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746992 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746995 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.746998 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747000 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747003 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747008 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747011 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747014 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747016 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747019 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747022 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747027 2582 flags.go:64] FLAG: --system-cgroups="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747030 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747035 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747038 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747040 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747044 2582 flags.go:64] FLAG: --tls-min-version="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747047 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747050 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747052 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747055 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747058 2582 flags.go:64] FLAG: --v="2" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747061 2582 flags.go:64] FLAG: --version="false" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747068 2582 flags.go:64] FLAG: --vmodule="" Jul 13 19:25:08.754230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747072 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.747075 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747199 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747203 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747206 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747209 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747212 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747214 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747219 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747222 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747226 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747229 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747231 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747235 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747238 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747240 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747243 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747246 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747248 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747252 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.754828 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747255 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747258 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747260 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747263 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747266 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747268 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747271 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747275 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747279 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747283 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747286 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747288 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747291 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747293 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747296 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747298 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747301 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747303 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747306 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.755340 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747308 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747311 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747313 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747316 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747318 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747321 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747324 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747327 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747329 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747332 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747334 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747337 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747340 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747343 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747352 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747354 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747357 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747359 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747362 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747364 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.755810 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747367 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747369 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747372 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747374 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747377 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747380 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747383 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747385 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747387 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747390 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747392 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747395 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747397 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747400 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747402 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747405 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747407 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747409 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747414 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747416 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.756528 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747419 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747434 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747437 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747440 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747443 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747446 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747448 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747451 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747454 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747457 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747459 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747462 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747464 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747467 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747469 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747472 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747474 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747477 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747479 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747482 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.757255 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747484 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.747487 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.748278 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.754962 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.754977 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.755818 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.755826 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756000 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756008 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756012 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756016 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756019 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756023 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756034 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756144 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.757765 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756156 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756162 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756168 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756173 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756179 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756184 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756189 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756195 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756201 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756207 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756212 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756227 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756235 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756240 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756245 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756252 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756257 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756265 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756273 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.758144 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756281 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756288 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756293 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756298 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756307 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756313 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756318 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756323 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756328 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756332 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756339 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756345 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756350 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756357 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756362 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756368 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756378 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756382 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756387 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756392 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.758660 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756397 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756401 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756406 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756410 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756415 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756420 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756441 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756445 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756450 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756460 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756465 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756469 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756474 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756479 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756483 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756487 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756492 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756496 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756501 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756507 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.759139 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756511 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756520 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756524 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756529 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756534 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756539 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756544 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756549 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756553 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756557 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756562 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756566 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756570 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756575 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756584 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756589 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756594 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756598 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756602 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756607 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.759628 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756611 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756616 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756621 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756626 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756630 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756635 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756644 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756648 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756653 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756658 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756662 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.756666 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.756675 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757316 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 19:25:08.760151 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757328 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757332 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757335 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757338 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757341 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757344 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757346 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757351 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757356 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757359 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757362 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757365 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757367 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757370 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757372 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757375 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757377 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757380 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757382 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757385 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 19:25:08.760512 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757388 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757391 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757393 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757396 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757399 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757402 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757406 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757409 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757413 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757415 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757418 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757439 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757443 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757446 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757449 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757452 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757454 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757457 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757459 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 19:25:08.760990 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757462 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757465 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757468 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757470 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757472 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757475 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757477 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757480 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757483 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757485 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757487 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757490 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757492 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757495 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757497 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757499 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757502 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757505 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757507 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 19:25:08.761561 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757510 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757512 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757515 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757518 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757520 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757523 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757525 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757527 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757530 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757532 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757534 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757537 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757539 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757541 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757544 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757546 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757548 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757550 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757553 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757555 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 19:25:08.762007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757557 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757560 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757562 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757565 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757568 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757572 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757575 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757577 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757580 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757582 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757585 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757587 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757590 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757592 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757595 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757597 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757600 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757603 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757605 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 19:25:08.762501 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:08.757607 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.757613 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.757739 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.759754 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.760992 2582 server.go:1019] "Starting client certificate rotation" Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.761095 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 19:25:08.762944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.761127 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 19:25:08.764067 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.764050 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 19:25:08.765202 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.765182 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 19:25:08.774690 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.774675 2582 log.go:25] "Validated CRI v1 runtime API" Jul 13 19:25:08.778826 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.778809 2582 log.go:25] "Validated CRI v1 image API" Jul 13 19:25:08.779626 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.779608 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 19:25:08.782037 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.782017 2582 fs.go:135] Filesystem UUIDs: map[3e6b4b1e-015d-41c1-85f1-e6d7654630ed:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 88861313-8fdd-46f6-8cc2-4a710329c2c5:/dev/nvme0n1p3] Jul 13 19:25:08.782100 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.782037 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 19:25:08.786975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.786865 2582 manager.go:216] Machine: {Timestamp:2026-07-13 19:25:08.785712419 +0000 UTC m=+0.347110454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2992168 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242fe72c856f9d2aa9b3db6802a240 SystemUUID:ec242fe7-2c85-6f9d-2aa9-b3db6802a240 BootID:c679b0dd-385a-40f0-ae9b-9df1e087bae3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:97:be:8e:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:97:be:8e:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:58:14:bb:b3:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 19:25:08.786975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.786974 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 19:25:08.787079 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.787054 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 19:25:08.788006 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.787981 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 19:25:08.788168 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.788011 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 19:25:08.788755 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.788743 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 19:25:08.788808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.788761 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 19:25:08.788808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.788772 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 19:25:08.790788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.790776 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 19:25:08.792176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.792163 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 19:25:08.793242 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.793227 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 19:25:08.795273 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795259 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 13 19:25:08.795405 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795394 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 19:25:08.795478 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795446 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 13 19:25:08.795478 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795466 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 19:25:08.795560 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795482 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 19:25:08.795560 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795495 2582 kubelet.go:398] "Adding apiserver pod source" Jul 13 19:25:08.795560 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795508 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 19:25:08.795984 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795967 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 19:25:08.796050 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.795996 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 19:25:08.799699 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.799658 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 19:25:08.801525 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801510 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 19:25:08.801594 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801547 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 19:25:08.801718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801708 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 19:25:08.801766 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801723 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 19:25:08.801766 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801729 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 19:25:08.801766 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801734 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 19:25:08.801766 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801742 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 19:25:08.801766 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.801748 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 19:25:08.802227 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802218 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 19:25:08.802227 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802227 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 19:25:08.802281 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802234 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 19:25:08.802281 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802239 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 19:25:08.802281 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802247 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 19:25:08.802281 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.802255 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 19:25:08.803735 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.803724 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 19:25:08.803796 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.803736 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 19:25:08.806580 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.806556 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 19:25:08.806653 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.806563 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 19:25:08.806653 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.806633 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 19:25:08.806952 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.806940 2582 server.go:1268] "Started kubelet" Jul 13 19:25:08.807043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.807018 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 19:25:08.807143 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.807094 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 19:25:08.807246 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.807161 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 19:25:08.807795 ip-10-0-135-83 systemd[1]: Started Kubernetes Kubelet. Jul 13 19:25:08.808408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.808340 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 19:25:08.809064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.809050 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 13 19:25:08.812137 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.812117 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 19:25:08.813093 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.812186 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-83.ec2.internal.18c1efe8642344fc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-83.ec2.internal,UID:ip-10-0-135-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-83.ec2.internal,},FirstTimestamp:2026-07-13 19:25:08.806911228 +0000 UTC m=+0.368309267,LastTimestamp:2026-07-13 19:25:08.806911228 +0000 UTC m=+0.368309267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-83.ec2.internal,}" Jul 13 19:25:08.813853 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.813839 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 19:25:08.813921 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.813839 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 19:25:08.814039 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.814007 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:08.814585 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814516 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 19:25:08.814585 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814532 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 19:25:08.814585 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814547 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 19:25:08.814756 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814651 2582 factory.go:55] Registering systemd factory Jul 13 19:25:08.814756 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814666 2582 factory.go:223] Registration of the systemd container factory successfully Jul 13 19:25:08.814756 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814672 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 13 19:25:08.814756 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814682 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 13 19:25:08.814926 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814851 2582 factory.go:153] Registering CRI-O factory Jul 13 19:25:08.814926 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814863 2582 factory.go:223] Registration of the crio container factory successfully Jul 13 19:25:08.814926 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814921 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 19:25:08.815048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814955 2582 factory.go:103] Registering Raw factory Jul 13 19:25:08.815048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.814972 2582 manager.go:1196] Started watching for new ooms in manager Jul 13 19:25:08.815687 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.815673 2582 manager.go:349] Starting recovery of all containers Jul 13 19:25:08.816211 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.816191 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 19:25:08.816332 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.816314 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 19:25:08.823638 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.823613 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 19:25:08.826211 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.826055 2582 manager.go:354] Recovery completed Jul 13 19:25:08.829095 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.829072 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 13 19:25:08.832116 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.832103 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:08.833849 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.833817 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:08.833911 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.833864 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:08.833911 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.833879 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:08.834385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.834368 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 19:25:08.834385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.834380 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 19:25:08.834527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.834385 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:08.834527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.834396 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 13 19:25:08.835050 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835033 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:08.835125 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835066 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:08.835125 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835079 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:08.835618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835605 2582 policy_none.go:49] "None policy: Start" Jul 13 19:25:08.835667 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835621 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 19:25:08.835667 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.835631 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 19:25:08.836751 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.836740 2582 policy_none.go:47] "Start" Jul 13 19:25:08.843394 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.843378 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5rm7n" Jul 13 19:25:08.844412 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.844349 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-83.ec2.internal.18c1efe865be5222 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-83.ec2.internal,UID:ip-10-0-135-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-83.ec2.internal,},FirstTimestamp:2026-07-13 19:25:08.83384989 +0000 UTC m=+0.395247924,LastTimestamp:2026-07-13 19:25:08.83384989 +0000 UTC m=+0.395247924,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-83.ec2.internal,}" Jul 13 19:25:08.853234 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.853217 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5rm7n" Jul 13 19:25:08.853987 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.853913 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-83.ec2.internal.18c1efe865bea62c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-83.ec2.internal,UID:ip-10-0-135-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-83.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-83.ec2.internal,},FirstTimestamp:2026-07-13 19:25:08.833871404 +0000 UTC m=+0.395269440,LastTimestamp:2026-07-13 19:25:08.833871404 +0000 UTC m=+0.395269440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-83.ec2.internal,}" Jul 13 19:25:08.877080 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877061 2582 manager.go:337] "Starting Device Plugin manager" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.877096 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877106 2582 server.go:87] "Starting device plugin registration server" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877384 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877396 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877593 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877690 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.877699 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.878103 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 19:25:08.881043 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.878142 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:08.954269 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.954237 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 19:25:08.954419 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.954285 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 19:25:08.954419 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.954309 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 19:25:08.954419 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.954346 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 19:25:08.956529 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.956506 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 19:25:08.978478 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.978459 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:08.979266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.979246 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:08.979363 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.979277 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:08.979363 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.979287 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:08.979363 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.979309 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-83.ec2.internal" Jul 13 19:25:08.988533 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:08.988483 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-83.ec2.internal" Jul 13 19:25:08.988533 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:08.988506 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-83.ec2.internal\": node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.010450 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.010418 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.055505 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.055477 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal"] Jul 13 19:25:09.055641 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.055549 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:09.057102 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.057088 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:09.057165 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.057119 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:09.057165 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.057131 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:09.058563 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.058551 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:09.058679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.058656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.058724 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.058689 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:09.059241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059208 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:09.059241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059239 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:09.059328 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059249 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:09.059328 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059213 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:09.059328 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059313 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:09.059328 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.059325 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:09.060468 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.060453 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.060537 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.060484 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:09.061113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.061097 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:09.061179 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.061130 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:09.061179 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.061144 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:09.075965 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.075944 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-83.ec2.internal\" not found" node="ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.080062 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.080042 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-83.ec2.internal\" not found" node="ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.110633 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.110612 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.115902 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.115887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.115969 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.115910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.115969 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.115928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b69fff8245d16238da7871546a70633-config\") pod \"kube-apiserver-proxy-ip-10-0-135-83.ec2.internal\" (UID: \"5b69fff8245d16238da7871546a70633\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.210913 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.210886 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.216201 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b69fff8245d16238da7871546a70633-config\") pod \"kube-apiserver-proxy-ip-10-0-135-83.ec2.internal\" (UID: \"5b69fff8245d16238da7871546a70633\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.216256 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.216256 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.216318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b69fff8245d16238da7871546a70633-config\") pod \"kube-apiserver-proxy-ip-10-0-135-83.ec2.internal\" (UID: \"5b69fff8245d16238da7871546a70633\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.216318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216282 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.216318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.216302 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe979ad5a4bace247ce4af4a756436ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal\" (UID: \"fe979ad5a4bace247ce4af4a756436ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.311667 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.311603 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.378114 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.378078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.383269 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.383249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:09.411999 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.411970 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.512523 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.512484 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.613025 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.612994 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.713494 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.713472 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.760858 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.760830 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 19:25:09.761316 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.760995 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 19:25:09.814194 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.814166 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.857068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.857033 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 19:20:09 +0000 UTC" deadline="2028-01-03 17:29:34.352307433 +0000 UTC" Jul 13 19:25:09.857068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.857066 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12934h4m24.495244874s" Jul 13 19:25:09.914418 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:09.914354 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:09.922089 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.922071 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 19:25:09.978897 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.978874 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 19:25:09.979943 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.979925 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientMemory" Jul 13 19:25:09.980021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.979959 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 19:25:09.980021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:09.979972 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeHasSufficientPID" Jul 13 19:25:10.014707 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.014690 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:10.100476 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:10.100443 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b69fff8245d16238da7871546a70633.slice/crio-d44254170245a2327a3ba565755f924d041702de7a216f46d1a6b0774c796271 WatchSource:0}: Error finding container d44254170245a2327a3ba565755f924d041702de7a216f46d1a6b0774c796271: Status 404 returned error can't find the container with id d44254170245a2327a3ba565755f924d041702de7a216f46d1a6b0774c796271 Jul 13 19:25:10.105661 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.105642 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 19:25:10.114910 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.114889 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-83.ec2.internal\" not found" Jul 13 19:25:10.173012 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.172934 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 19:25:10.214679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.214654 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" Jul 13 19:25:10.225503 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.225482 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 19:25:10.227306 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.227293 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" Jul 13 19:25:10.239484 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.239465 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 19:25:10.389336 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.389308 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 19:25:10.496202 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:10.496168 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe979ad5a4bace247ce4af4a756436ce.slice/crio-9120bb3cfa093ad74fd67ecd1329a24a3b29824423d49b510071a8b6c3764467 WatchSource:0}: Error finding container 9120bb3cfa093ad74fd67ecd1329a24a3b29824423d49b510071a8b6c3764467: Status 404 returned error can't find the container with id 9120bb3cfa093ad74fd67ecd1329a24a3b29824423d49b510071a8b6c3764467 Jul 13 19:25:10.796465 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.796353 2582 apiserver.go:52] "Watching apiserver" Jul 13 19:25:10.804747 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.804718 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 19:25:10.805193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.805156 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qb8q5","kube-system/konnectivity-agent-rpmz9","openshift-cluster-node-tuning-operator/tuned-mhwcw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal","openshift-multus/multus-additional-cni-plugins-5dkxv","openshift-multus/multus-wrcgk","openshift-multus/network-metrics-daemon-9qx6r","openshift-network-diagnostics/network-check-target-dhhmb","kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz","openshift-dns/node-resolver-t762r","openshift-image-registry/node-ca-ccvgs","openshift-network-operator/iptables-alerter-tw2g8","openshift-ovn-kubernetes/ovnkube-node-q55l7"] Jul 13 19:25:10.806925 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.806898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:10.807031 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.806997 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:10.808743 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.808716 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.810624 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.810604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.811436 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811397 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 19:25:10.811551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811420 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.811551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811481 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.811750 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.811921 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811901 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9g4cz\"" Jul 13 19:25:10.811988 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.811958 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 19:25:10.812204 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.812187 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 19:25:10.812283 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.812266 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 19:25:10.813041 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.813022 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 19:25:10.813124 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.813042 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.813345 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.813325 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckp9l\"" Jul 13 19:25:10.813422 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.813368 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.813620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.813601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814236 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814271 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814332 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwpk8\"" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814415 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814438 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.814995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.814647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.815931 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.815912 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.816186 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.816152 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vntvl\"" Jul 13 19:25:10.816400 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.816383 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.816634 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.816617 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 19:25:10.816995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.816966 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 19:25:10.817186 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.817171 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 19:25:10.818954 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.817533 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 19:25:10.818954 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.817577 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5mptp\"" Jul 13 19:25:10.820661 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.820366 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.821682 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.821656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:10.821778 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.821716 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:10.823176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823022 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.823176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823063 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f8nv8\"" Jul 13 19:25:10.823176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823137 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 19:25:10.823176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823148 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823346 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-os-release\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-etc-kubernetes\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-bin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovn-node-metrics-cert\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.823500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-system-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-kubelet\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxm8v\" (UniqueName: \"kubernetes.io/projected/c07c93c3-afc5-44b6-a043-702914d1a37e-kube-api-access-cxm8v\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-slash\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-etc-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-ovn\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:10.823805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gs4\" (UniqueName: \"kubernetes.io/projected/33ea8156-02d6-4487-8148-d6c4c28fa8e0-kube-api-access-m6gs4\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-daemon-config\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-kubelet\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-os-release\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwq6t\" (UniqueName: \"kubernetes.io/projected/be5af7b1-2201-4646-967f-b7e6d1795a6c-kube-api-access-vwq6t\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.823989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-systemd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824036 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdq7g\" (UniqueName: \"kubernetes.io/projected/8d1483bf-572f-4e1c-8f18-d380a09f366f-kube-api-access-kdq7g\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-socket-dir-parent\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-multus-certs\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-node-log\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-netd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-conf-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-netns\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.824266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9m4\" (UniqueName: \"kubernetes.io/projected/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-kube-api-access-2l9m4\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d1483bf-572f-4e1c-8f18-d380a09f366f-host-slash\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-cni-binary-copy\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824871 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-netns\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.824941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-registration-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.825058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-systemd-units\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.825452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-script-lib\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.825452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cnibin\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.825452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfc9\" (UniqueName: \"kubernetes.io/projected/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-kube-api-access-csfc9\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.825452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825357 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-multus\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.825452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-log-socket\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.825714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-env-overrides\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.825916 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-device-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.826033 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-var-lib-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.826033 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.825972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-socket-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.826151 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.826151 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33ea8156-02d6-4487-8148-d6c4c28fa8e0-host\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.826151 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33ea8156-02d6-4487-8148-d6c4c28fa8e0-serviceca\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.826299 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.826168 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:10.826299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.826299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826232 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.826299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-cnibin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-k8s-cni-cncf-io\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-sys-fs\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-config\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d1483bf-572f-4e1c-8f18-d380a09f366f-iptables-alerter-script\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-system-cni-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-hostroot\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.826679 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.826621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-bin\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.829423 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.829399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:10.831683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.831659 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.832149 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.832125 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.832716 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.832691 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldd2f\"" Jul 13 19:25:10.832716 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.832711 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.833213 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.833194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:10.834378 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.834355 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 19:25:10.834488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.834466 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 19:25:10.834624 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.834598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8xmwg\"" Jul 13 19:25:10.835712 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.835685 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 19:25:10.835808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.835737 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-67wxc\"" Jul 13 19:25:10.836014 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.835961 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 19:25:10.856113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.856089 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h2v97" Jul 13 19:25:10.864219 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.864200 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h2v97" Jul 13 19:25:10.915880 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.915852 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 19:25:10.927437 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927391 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-var-lib-kubelet\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-device-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-var-lib-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927514 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-kubernetes\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-systemd\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.927591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-konnectivity-ca\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-device-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-socket-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-var-lib-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33ea8156-02d6-4487-8148-d6c4c28fa8e0-host\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33ea8156-02d6-4487-8148-d6c4c28fa8e0-serviceca\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-socket-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927791 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33ea8156-02d6-4487-8148-d6c4c28fa8e0-host\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927842 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-cnibin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-k8s-cni-cncf-io\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.927913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-k8s-cni-cncf-io\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-sys-fs\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-config\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d1483bf-572f-4e1c-8f18-d380a09f366f-iptables-alerter-script\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-sys-fs\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-system-cni-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.927977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-cnibin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-hostroot\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928086 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-dbus\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-run\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-tuned\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-bin\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-os-release\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-etc-kubernetes\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-bin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.928618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33ea8156-02d6-4487-8148-d6c4c28fa8e0-serviceca\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-kubelet-config\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-tmp\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-agent-certs\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovn-node-metrics-cert\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-system-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-kubelet\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928532 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-system-cni-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxm8v\" (UniqueName: \"kubernetes.io/projected/c07c93c3-afc5-44b6-a043-702914d1a37e-kube-api-access-cxm8v\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d1483bf-572f-4e1c-8f18-d380a09f366f-iptables-alerter-script\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928565 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffmg\" (UniqueName: \"kubernetes.io/projected/d6cbfa7b-786e-425b-82e5-10d3faaf9790-kube-api-access-cffmg\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-hostroot\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-modprobe-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-bin\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-bin\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.929445 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-slash\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928663 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-etc-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928672 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-slash\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-sys\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58c8\" (UniqueName: \"kubernetes.io/projected/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-kube-api-access-w58c8\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-ovn\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-config\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gs4\" (UniqueName: \"kubernetes.io/projected/33ea8156-02d6-4487-8148-d6c4c28fa8e0-kube-api-access-m6gs4\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-daemon-config\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928914 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-kubelet\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928934 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-kubelet\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-etc-kubernetes\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.930303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-system-cni-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.928960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-os-release\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-ovn\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929011 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929038 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-os-release\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929089 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-os-release\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-etc-openvswitch\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-kubelet\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s85\" (UniqueName: \"kubernetes.io/projected/3a9254d4-7197-48a0-a784-bb398df08092-kube-api-access-h5s85\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwq6t\" (UniqueName: \"kubernetes.io/projected/be5af7b1-2201-4646-967f-b7e6d1795a6c-kube-api-access-vwq6t\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-systemd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdq7g\" (UniqueName: \"kubernetes.io/projected/8d1483bf-572f-4e1c-8f18-d380a09f366f-kube-api-access-kdq7g\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-socket-dir-parent\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-multus-certs\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-node-log\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-netd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-conf-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-conf\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-netns\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9m4\" (UniqueName: \"kubernetes.io/projected/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-kube-api-access-2l9m4\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.929992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-lib-modules\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d1483bf-572f-4e1c-8f18-d380a09f366f-host-slash\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-cni-binary-copy\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-netns\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6cbfa7b-786e-425b-82e5-10d3faaf9790-hosts-file\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.931774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-registration-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-systemd-units\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-script-lib\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cnibin\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfc9\" (UniqueName: \"kubernetes.io/projected/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-kube-api-access-csfc9\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-multus\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.930997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cbfa7b-786e-425b-82e5-10d3faaf9790-tmp-dir\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-daemon-config\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysconfig\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-host\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-socket-dir-parent\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931517 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-node-log\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-log-socket\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-cni-netd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-env-overrides\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931663 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-multus-conf-dir\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.932315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-multus-certs\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5af7b1-2201-4646-967f-b7e6d1795a6c-registration-dir\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.931967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-systemd-units\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932188 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-env-overrides\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d1483bf-572f-4e1c-8f18-d380a09f366f-host-slash\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932387 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-var-lib-cni-multus\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932475 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-cnibin\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovnkube-script-lib\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-run-systemd\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932736 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c07c93c3-afc5-44b6-a043-702914d1a37e-host-run-netns\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-host-run-netns\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933112 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.932839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-log-socket\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933633 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.933262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-ovn-node-metrics-cert\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.933633 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.933538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c07c93c3-afc5-44b6-a043-702914d1a37e-cni-binary-copy\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.935351 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.935328 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:10.935351 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.935354 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:10.935567 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.935376 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:10.935567 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:10.935474 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:11.435433112 +0000 UTC m=+2.996831161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:10.938060 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.938034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxm8v\" (UniqueName: \"kubernetes.io/projected/c07c93c3-afc5-44b6-a043-702914d1a37e-kube-api-access-cxm8v\") pod \"multus-wrcgk\" (UID: \"c07c93c3-afc5-44b6-a043-702914d1a37e\") " pod="openshift-multus/multus-wrcgk" Jul 13 19:25:10.938576 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.938551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gs4\" (UniqueName: \"kubernetes.io/projected/33ea8156-02d6-4487-8148-d6c4c28fa8e0-kube-api-access-m6gs4\") pod \"node-ca-ccvgs\" (UID: \"33ea8156-02d6-4487-8148-d6c4c28fa8e0\") " pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:10.938998 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.938978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwq6t\" (UniqueName: \"kubernetes.io/projected/be5af7b1-2201-4646-967f-b7e6d1795a6c-kube-api-access-vwq6t\") pod \"aws-ebs-csi-driver-node-hwwfz\" (UID: \"be5af7b1-2201-4646-967f-b7e6d1795a6c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:10.942266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.942244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdq7g\" (UniqueName: \"kubernetes.io/projected/8d1483bf-572f-4e1c-8f18-d380a09f366f-kube-api-access-kdq7g\") pod \"iptables-alerter-tw2g8\" (UID: \"8d1483bf-572f-4e1c-8f18-d380a09f366f\") " pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:10.945971 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.945949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfc9\" (UniqueName: \"kubernetes.io/projected/99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb-kube-api-access-csfc9\") pod \"multus-additional-cni-plugins-5dkxv\" (UID: \"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb\") " pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:10.947288 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.947267 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9m4\" (UniqueName: \"kubernetes.io/projected/733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea-kube-api-access-2l9m4\") pod \"ovnkube-node-q55l7\" (UID: \"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:10.959154 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.959103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" event={"ID":"fe979ad5a4bace247ce4af4a756436ce","Type":"ContainerStarted","Data":"9120bb3cfa093ad74fd67ecd1329a24a3b29824423d49b510071a8b6c3764467"} Jul 13 19:25:10.960364 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:10.960334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" event={"ID":"5b69fff8245d16238da7871546a70633","Type":"ContainerStarted","Data":"d44254170245a2327a3ba565755f924d041702de7a216f46d1a6b0774c796271"} Jul 13 19:25:11.030234 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.030203 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 19:25:11.032224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6cbfa7b-786e-425b-82e5-10d3faaf9790-hosts-file\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.032297 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cbfa7b-786e-425b-82e5-10d3faaf9790-tmp-dir\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.032297 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysconfig\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032297 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-host\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-var-lib-kubelet\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-kubernetes\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-systemd\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-konnectivity-ca\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:11.032442 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6cbfa7b-786e-425b-82e5-10d3faaf9790-hosts-file\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-var-lib-kubelet\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-dbus\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysconfig\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-systemd\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.032457 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-host\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032577 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-dbus\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.032627 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:11.532604734 +0000 UTC m=+3.094002785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cbfa7b-786e-425b-82e5-10d3faaf9790-tmp-dir\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-kubernetes\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-run\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-tuned\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-run\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.032737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-kubelet-config\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-tmp\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-agent-certs\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffmg\" (UniqueName: \"kubernetes.io/projected/d6cbfa7b-786e-425b-82e5-10d3faaf9790-kube-api-access-cffmg\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d918169a-694f-4831-b7f0-9844aa3acecd-kubelet-config\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-modprobe-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-sys\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58c8\" (UniqueName: \"kubernetes.io/projected/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-kube-api-access-w58c8\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s85\" (UniqueName: \"kubernetes.io/projected/3a9254d4-7197-48a0-a784-bb398df08092-kube-api-access-h5s85\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.032999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-sys\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-konnectivity-ca\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-conf\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-lib-modules\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-lib-modules\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-conf\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-modprobe-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.033495 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.033297 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:11.034610 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.033347 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:11.533327401 +0000 UTC m=+3.094725428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:11.034610 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.033459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-sysctl-d\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.035318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.035216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-etc-tuned\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.035912 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.035886 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cc9f40-bab1-4714-83dd-bd8e75e89a92-agent-certs\") pod \"konnectivity-agent-rpmz9\" (UID: \"a1cc9f40-bab1-4714-83dd-bd8e75e89a92\") " pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:11.036034 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.035989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-tmp\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.053687 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.053615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58c8\" (UniqueName: \"kubernetes.io/projected/c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd-kube-api-access-w58c8\") pod \"tuned-mhwcw\" (UID: \"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd\") " pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.053886 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.053850 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s85\" (UniqueName: \"kubernetes.io/projected/3a9254d4-7197-48a0-a784-bb398df08092-kube-api-access-h5s85\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:11.054002 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.053909 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffmg\" (UniqueName: \"kubernetes.io/projected/d6cbfa7b-786e-425b-82e5-10d3faaf9790-kube-api-access-cffmg\") pod \"node-resolver-t762r\" (UID: \"d6cbfa7b-786e-425b-82e5-10d3faaf9790\") " pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.123616 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.123574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:11.134222 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.134010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tw2g8" Jul 13 19:25:11.145241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.145217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccvgs" Jul 13 19:25:11.152217 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.152198 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" Jul 13 19:25:11.159360 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.159326 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wrcgk" Jul 13 19:25:11.168644 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.168487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" Jul 13 19:25:11.176523 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.176495 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t762r" Jul 13 19:25:11.184628 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.184599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" Jul 13 19:25:11.192783 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.192762 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:11.437321 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.437276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:11.437533 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.437429 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:11.437533 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.437456 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:11.437533 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.437470 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:11.437708 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.437568 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:12.437544534 +0000 UTC m=+3.998942571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:11.537979 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.537946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:11.538143 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.537995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:11.538143 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.538111 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:11.538143 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.538111 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:11.538275 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.538181 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:12.538162481 +0000 UTC m=+4.099560505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:11.538275 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.538199 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:12.538190398 +0000 UTC m=+4.099588423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:11.866904 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.866862 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 19:20:11 +0000 UTC" deadline="2027-12-24 05:47:18.299397557 +0000 UTC" Jul 13 19:25:11.866904 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.866897 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12682h22m6.432504711s" Jul 13 19:25:11.955255 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:11.955219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:11.955427 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:11.955340 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:12.164522 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:12.164492 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733e4d2b_0d31_46ae_ac4c_ea4b0d9763ea.slice/crio-87b2ccb9c59c16e0796b4290ffdfe8b6538bfa9aad2f44a381e0ee67241e124d WatchSource:0}: Error finding container 87b2ccb9c59c16e0796b4290ffdfe8b6538bfa9aad2f44a381e0ee67241e124d: Status 404 returned error can't find the container with id 87b2ccb9c59c16e0796b4290ffdfe8b6538bfa9aad2f44a381e0ee67241e124d Jul 13 19:25:12.166777 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:12.166745 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ea8156_02d6_4487_8148_d6c4c28fa8e0.slice/crio-c296f8dbf86849f2112da3f8b2ad5b21f88b19772bb47708960dbda08b73ca8d WatchSource:0}: Error finding container c296f8dbf86849f2112da3f8b2ad5b21f88b19772bb47708960dbda08b73ca8d: Status 404 returned error can't find the container with id c296f8dbf86849f2112da3f8b2ad5b21f88b19772bb47708960dbda08b73ca8d Jul 13 19:25:12.167729 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:12.167698 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1483bf_572f_4e1c_8f18_d380a09f366f.slice/crio-d7776e9967341814fe331910cf3fcfe0f6d2bf0f398817e0125ae5ee502b6e9e WatchSource:0}: Error finding container d7776e9967341814fe331910cf3fcfe0f6d2bf0f398817e0125ae5ee502b6e9e: Status 404 returned error can't find the container with id d7776e9967341814fe331910cf3fcfe0f6d2bf0f398817e0125ae5ee502b6e9e Jul 13 19:25:12.446903 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.446720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:12.446903 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.446892 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:12.447068 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.446923 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:12.447068 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.446937 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:12.447068 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.447006 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:14.446985118 +0000 UTC m=+6.008383145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:12.548230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.548190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:12.548358 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.548265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:12.548358 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.548321 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:12.548461 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.548364 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:12.548461 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.548405 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:14.548387695 +0000 UTC m=+6.109785717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:12.548461 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.548420 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:14.548412938 +0000 UTC m=+6.109810962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:12.867105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.867044 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 19:20:11 +0000 UTC" deadline="2028-01-29 02:08:52.984621159 +0000 UTC" Jul 13 19:25:12.867105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.867100 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13542h43m40.11752501s" Jul 13 19:25:12.956996 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.956956 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:12.957183 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.957117 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:12.957701 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.957631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:12.957832 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:12.957729 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:12.967717 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.967245 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" event={"ID":"5b69fff8245d16238da7871546a70633","Type":"ContainerStarted","Data":"143012dea60e2922d7ca0ffbbf90080348a6eac1cddaf50090fb6d5321931578"} Jul 13 19:25:12.969158 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.968728 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t762r" event={"ID":"d6cbfa7b-786e-425b-82e5-10d3faaf9790","Type":"ContainerStarted","Data":"77170cc8bc9866e0f5cb34eed67544cfa1a671e89a3ea5ffb41d50fb7079628c"} Jul 13 19:25:12.970537 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.970511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpmz9" event={"ID":"a1cc9f40-bab1-4714-83dd-bd8e75e89a92","Type":"ContainerStarted","Data":"437ca2ba59e076ebd4210c6cc291a2b02c90cfe1fed443e576e9438667e2a283"} Jul 13 19:25:12.972514 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.972487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tw2g8" event={"ID":"8d1483bf-572f-4e1c-8f18-d380a09f366f","Type":"ContainerStarted","Data":"d7776e9967341814fe331910cf3fcfe0f6d2bf0f398817e0125ae5ee502b6e9e"} Jul 13 19:25:12.975610 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.975551 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccvgs" event={"ID":"33ea8156-02d6-4487-8148-d6c4c28fa8e0","Type":"ContainerStarted","Data":"c296f8dbf86849f2112da3f8b2ad5b21f88b19772bb47708960dbda08b73ca8d"} Jul 13 19:25:12.977604 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.977577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"87b2ccb9c59c16e0796b4290ffdfe8b6538bfa9aad2f44a381e0ee67241e124d"} Jul 13 19:25:12.981107 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.981075 2582 generic.go:356] "Generic (PLEG): container finished" podID="fe979ad5a4bace247ce4af4a756436ce" containerID="4569eedbaf66e841214731c2f46e503c6a7db6fdea029a86180c83f2f7636e04" exitCode=0 Jul 13 19:25:12.981397 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.981372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" event={"ID":"fe979ad5a4bace247ce4af4a756436ce","Type":"ContainerDied","Data":"4569eedbaf66e841214731c2f46e503c6a7db6fdea029a86180c83f2f7636e04"} Jul 13 19:25:12.984251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.983847 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-83.ec2.internal" podStartSLOduration=1.983828209 podStartE2EDuration="1.983828209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:12.982947889 +0000 UTC m=+4.544345937" watchObservedRunningTime="2026-07-13 19:25:12.983828209 +0000 UTC m=+4.545226264" Jul 13 19:25:12.984885 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.984859 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" event={"ID":"be5af7b1-2201-4646-967f-b7e6d1795a6c","Type":"ContainerStarted","Data":"3d345c5d2c001e8c28b6d40f6666b0356748498b505198e11cc2b9fcb394f03f"} Jul 13 19:25:12.987816 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.987795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wrcgk" event={"ID":"c07c93c3-afc5-44b6-a043-702914d1a37e","Type":"ContainerStarted","Data":"99a9faaa446c51ba9338662aea5954faa77ec193d6222f9a5ca903f5b3b8d58b"} Jul 13 19:25:12.995325 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.995287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerStarted","Data":"c09335710177dae60edfb65ab6875e73f0af3be5783818a9e640f0a389d0e11e"} Jul 13 19:25:12.999698 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:12.999672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" event={"ID":"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd","Type":"ContainerStarted","Data":"24078527410cb9862896ed5a90deffbc639a6599c8dd087e93bd115fbb2fe894"} Jul 13 19:25:13.954920 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:13.954885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:13.955495 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:13.955021 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:14.037150 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.037104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" event={"ID":"fe979ad5a4bace247ce4af4a756436ce","Type":"ContainerStarted","Data":"db4c64e20afa588ebaf549723d9c8afd73109242486a2d3eab308097c59c3b46"} Jul 13 19:25:14.053749 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.053694 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-83.ec2.internal" podStartSLOduration=3.053671796 podStartE2EDuration="3.053671796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:14.053491818 +0000 UTC m=+5.614889870" watchObservedRunningTime="2026-07-13 19:25:14.053671796 +0000 UTC m=+5.615069846" Jul 13 19:25:14.466598 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.466554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:14.466818 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.466740 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:14.466818 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.466767 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:14.466818 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.466783 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:14.466993 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.466847 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:18.46682737 +0000 UTC m=+10.028225415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:14.567261 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.567218 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:14.567466 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.567320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:14.567565 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.567483 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:14.567565 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.567550 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:18.567530745 +0000 UTC m=+10.128928784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:14.567993 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.567971 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:14.568086 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.568026 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:18.568010363 +0000 UTC m=+10.129408391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:14.955858 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.955608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:14.955858 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:14.955619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:14.955858 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.955759 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:14.955858 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:14.955834 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:15.955066 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:15.955025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:15.955281 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:15.955169 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:16.955668 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:16.955631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:16.956155 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:16.955631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:16.956155 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:16.955801 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:16.956155 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:16.955891 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:17.955246 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:17.955109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:17.955246 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:17.955239 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:18.502888 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:18.502838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:18.503440 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.503078 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:18.503440 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.503104 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:18.503440 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.503118 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:18.503440 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.503191 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:26.503168986 +0000 UTC m=+18.064567011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:18.604302 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:18.604261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:18.604483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:18.604357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:18.604483 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.604469 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:18.604574 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.604514 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:18.604574 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.604528 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:26.604511864 +0000 UTC m=+18.165909888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:18.604648 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.604596 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:26.60457533 +0000 UTC m=+18.165973360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:18.955958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:18.955926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:18.956182 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.956054 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:18.956182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:18.956131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:18.956284 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:18.956244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:19.956056 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:19.955525 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:19.956056 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:19.955672 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:20.955546 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:20.955510 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:20.955749 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:20.955670 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:20.956119 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:20.956093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:20.956639 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:20.956206 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:21.955530 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:21.955496 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:21.955736 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:21.955598 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:22.954848 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:22.954816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:22.955283 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:22.954856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:22.955283 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:22.954943 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:22.955283 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:22.955170 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:23.955213 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:23.955176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:23.955713 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:23.955295 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:24.954753 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:24.954715 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:24.954944 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:24.954865 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:24.954944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:24.954932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:24.955077 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:24.955052 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:25.955003 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:25.954969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:25.955412 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:25.955080 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:26.574019 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:26.573986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:26.574190 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.574140 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:26.574190 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.574159 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:26.574190 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.574170 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:26.574349 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.574232 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:42.574214646 +0000 UTC m=+34.135612670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:26.674982 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:26.674928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:26.675163 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:26.675043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:26.675163 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.675101 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:26.675163 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.675150 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:26.675311 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.675181 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:25:42.675158404 +0000 UTC m=+34.236556442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:26.675311 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.675203 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:42.675188796 +0000 UTC m=+34.236586832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:26.954711 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:26.954673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:26.954872 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:26.954726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:26.954872 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.954793 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:26.955050 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:26.954919 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:27.955525 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:27.955489 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:27.955974 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:27.955608 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:28.956448 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:28.955995 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:28.956448 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:28.956126 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:28.956448 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:28.956192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:28.956448 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:28.956310 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:29.955259 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:29.955221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:29.955459 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:29.955353 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:30.954944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:30.954903 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:30.954944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:30.954946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:30.955536 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:30.955051 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:30.955536 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:30.955158 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:31.955790 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:31.955157 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:31.955790 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:31.955475 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:32.083881 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.083790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpmz9" event={"ID":"a1cc9f40-bab1-4714-83dd-bd8e75e89a92","Type":"ContainerStarted","Data":"15991886b31f77444c5bbae77900109746f993ae9de32b713f5a8c009754c88e"} Jul 13 19:25:32.085199 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.085161 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccvgs" event={"ID":"33ea8156-02d6-4487-8148-d6c4c28fa8e0","Type":"ContainerStarted","Data":"5643fd10a2cd9cebafb62ad8a92547603bbafabaa5602b52dd38172290897abd"} Jul 13 19:25:32.086480 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.086458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerStarted","Data":"504cf434b358a748ca5f8eb9d74fdc5a2358107ca723d3f746193d60747a648f"} Jul 13 19:25:32.087745 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.087725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" event={"ID":"c5a5d0f8-7c10-48a1-a8a9-d52b03f7a9bd","Type":"ContainerStarted","Data":"0e8e167ef2972d69421f38294c54ccd8fd15e027b0046d74d7f0e31a1be47724"} Jul 13 19:25:32.113700 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.113661 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rpmz9" podStartSLOduration=2.474195066 podStartE2EDuration="22.11364699s" totalImagesPullingTime="19.639451924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:32.097987971 +0000 UTC m=+23.659386015" watchObservedRunningTime="2026-07-13 19:25:32.11364699 +0000 UTC m=+23.675045034" Jul 13 19:25:32.113805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.113741 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mhwcw" podStartSLOduration=2.473481435 podStartE2EDuration="22.113735258s" totalImagesPullingTime="19.640253823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:32.113158683 +0000 UTC m=+23.674556727" watchObservedRunningTime="2026-07-13 19:25:32.113735258 +0000 UTC m=+23.675133302" Jul 13 19:25:32.127046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.127010 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ccvgs" podStartSLOduration=3.464117125 podStartE2EDuration="23.126996065s" totalImagesPullingTime="19.66287894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:32.126536422 +0000 UTC m=+23.687934477" watchObservedRunningTime="2026-07-13 19:25:32.126996065 +0000 UTC m=+23.688394109" Jul 13 19:25:32.955495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.955240 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:32.955676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:32.955271 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:32.955676 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:32.955631 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:32.955802 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:32.955669 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:33.091608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.091568 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t762r" event={"ID":"d6cbfa7b-786e-425b-82e5-10d3faaf9790","Type":"ContainerStarted","Data":"11ae202806d20d7c4f3547c4306431574142b2079dd87240310264fa10d9e041"} Jul 13 19:25:33.094742 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.094718 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:25:33.095137 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095111 2582 generic.go:356] "Generic (PLEG): container finished" podID="733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea" containerID="d521876f6988e40ec3291a4de87d26bd50ea72e72ea73c842c27f221841dda46" exitCode=1 Jul 13 19:25:33.095263 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"f3cbad56f23905baca93581a19086be78977ef4966e6d5dd19d941caf73be5fd"} Jul 13 19:25:33.095329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"29cd50d98d8693eb7f004038f05e3163fa5dd26265a74aa4908c952afacec190"} Jul 13 19:25:33.095329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095322 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"f7aa679e1542ad0cbdfff77b37d0d2a29d10b25075b73e3c520101af45822793"} Jul 13 19:25:33.095456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"a7bd22f8c6c24d22c84680929a889c073d093c4cc9443092c22450dd302f4916"} Jul 13 19:25:33.095456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerDied","Data":"d521876f6988e40ec3291a4de87d26bd50ea72e72ea73c842c27f221841dda46"} Jul 13 19:25:33.095559 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.095490 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"0508a8c2ad5045dc7bf5e182912a6a2eb79599ded71dbea5091613892a2ce465"} Jul 13 19:25:33.097199 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.097173 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" event={"ID":"be5af7b1-2201-4646-967f-b7e6d1795a6c","Type":"ContainerStarted","Data":"46114a60178beacf74ba2552a898b9e483136ca75a94d3060248fc0598e15e05"} Jul 13 19:25:33.098719 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.098697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wrcgk" event={"ID":"c07c93c3-afc5-44b6-a043-702914d1a37e","Type":"ContainerStarted","Data":"19802a023af8ccce78f3090a343170844f138ecd051bc60dac7339c90ca10ece"} Jul 13 19:25:33.100232 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.100212 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="504cf434b358a748ca5f8eb9d74fdc5a2358107ca723d3f746193d60747a648f" exitCode=0 Jul 13 19:25:33.100333 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.100300 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"504cf434b358a748ca5f8eb9d74fdc5a2358107ca723d3f746193d60747a648f"} Jul 13 19:25:33.107492 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.107457 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t762r" podStartSLOduration=3.475277734 podStartE2EDuration="23.107445219s" totalImagesPullingTime="19.632167485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:33.106830015 +0000 UTC m=+24.668228077" watchObservedRunningTime="2026-07-13 19:25:33.107445219 +0000 UTC m=+24.668843259" Jul 13 19:25:33.129347 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.129305 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wrcgk" podStartSLOduration=4.44025692 podStartE2EDuration="24.129292089s" totalImagesPullingTime="19.689035169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:33.129094294 +0000 UTC m=+24.690492363" watchObservedRunningTime="2026-07-13 19:25:33.129292089 +0000 UTC m=+24.690690133" Jul 13 19:25:33.684176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.684150 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 19:25:33.897622 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.897533 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T19:25:33.684172602Z","UUID":"694f8688-eb14-4f56-ba1a-f4df8aa7d711","Handler":null,"Name":"","Endpoint":""} Jul 13 19:25:33.898920 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.898902 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 19:25:33.899021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.898934 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 19:25:33.954935 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:33.954907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:33.955104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:33.955080 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:34.104138 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.104109 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:34.104611 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.104340 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tw2g8" event={"ID":"8d1483bf-572f-4e1c-8f18-d380a09f366f","Type":"ContainerStarted","Data":"7da9966bb3f1e32a3130cf7f123c38fe2a4be4b3feab95ff6552c347e4d47a56"} Jul 13 19:25:34.104877 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.104853 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:34.107510 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.107477 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" event={"ID":"be5af7b1-2201-4646-967f-b7e6d1795a6c","Type":"ContainerStarted","Data":"0050d67e5001fc501288e3d29b3d50afa11ce30888773682976e7fb439740276"} Jul 13 19:25:34.134358 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.134309 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tw2g8" podStartSLOduration=5.469706682 podStartE2EDuration="25.134290345s" totalImagesPullingTime="19.664583663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:34.119939325 +0000 UTC m=+25.681337386" watchObservedRunningTime="2026-07-13 19:25:34.134290345 +0000 UTC m=+25.695688390" Jul 13 19:25:34.955500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.955235 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:34.955665 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:34.955235 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:34.955665 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:34.955620 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:34.955776 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:34.955686 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:35.113575 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.113544 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:25:35.114087 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.114059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"df292645ded3c27e7fd11a630559e7f83d3f4d341662bc72c03f39e238b78b75"} Jul 13 19:25:35.116289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.116259 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" event={"ID":"be5af7b1-2201-4646-967f-b7e6d1795a6c","Type":"ContainerStarted","Data":"b1c73056578c795e6e8f9323843b560930dce94d963f65e48f0f27eb325f8f02"} Jul 13 19:25:35.116289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.116284 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 13 19:25:35.134911 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.134867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwwfz" podStartSLOduration=3.466960393 podStartE2EDuration="25.134852121s" totalImagesPullingTime="21.667891728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:35.134347823 +0000 UTC m=+26.695745869" watchObservedRunningTime="2026-07-13 19:25:35.134852121 +0000 UTC m=+26.696250164" Jul 13 19:25:35.955136 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:35.955099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:35.955390 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:35.955204 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:36.955497 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:36.955462 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:36.955907 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:36.955471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:36.955907 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:36.955615 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:36.955907 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:36.955677 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:37.954673 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:37.954487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:37.954825 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:37.954743 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:38.126403 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.126376 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:25:38.127105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.126742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"b7438fea05301b439821cc5ca974cd612b3d92e510ff9bb8f994da8fa4f9d546"} Jul 13 19:25:38.127105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.127006 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:38.127211 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.127157 2582 scope.go:117] "RemoveContainer" containerID="d521876f6988e40ec3291a4de87d26bd50ea72e72ea73c842c27f221841dda46" Jul 13 19:25:38.128736 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.128713 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="4d93a8be9960d8127e1e4c796da1c63dcd669d37604708358c39d4b108e8bbac" exitCode=0 Jul 13 19:25:38.128852 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.128749 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"4d93a8be9960d8127e1e4c796da1c63dcd669d37604708358c39d4b108e8bbac"} Jul 13 19:25:38.144462 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.144439 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:38.956279 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.956243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:38.956483 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:38.956359 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:38.956483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:38.956412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:38.956575 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:38.956514 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:39.139775 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.139747 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:25:39.140408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.140382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" event={"ID":"733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea","Type":"ContainerStarted","Data":"0775c69cc8c8e97abcf9a361d3fb044ace6429db6f5725b6c67ba1b8fc978b1d"} Jul 13 19:25:39.140788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.140766 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:39.140888 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.140846 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:39.158849 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.158823 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:25:39.169206 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.169161 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" podStartSLOduration=10.115413959 podStartE2EDuration="30.169145505s" totalImagesPullingTime="20.053731546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:39.167608671 +0000 UTC m=+30.729006716" watchObservedRunningTime="2026-07-13 19:25:39.169145505 +0000 UTC m=+30.730543557" Jul 13 19:25:39.594380 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.594345 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qx6r"] Jul 13 19:25:39.594552 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.594460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:39.594624 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:39.594550 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:39.597801 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.597764 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qb8q5"] Jul 13 19:25:39.597973 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.597864 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:39.597973 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:39.597948 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:39.598547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.598524 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dhhmb"] Jul 13 19:25:39.598633 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:39.598619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:39.598720 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:39.598701 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:40.147646 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:40.147608 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="af3ca53b486a0577334fbb880f902928f0e3a30fd7ab94213f134c6b1a928c38" exitCode=0 Jul 13 19:25:40.148013 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:40.147686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"af3ca53b486a0577334fbb880f902928f0e3a30fd7ab94213f134c6b1a928c38"} Jul 13 19:25:40.954868 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:40.954642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:40.954966 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:40.954655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:40.955015 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:40.954986 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:40.955055 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:40.954687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:40.955105 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:40.955066 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:40.955172 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:40.955147 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:41.033978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:41.033944 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:41.034106 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:41.034094 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 13 19:25:41.034526 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:41.034502 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rpmz9" Jul 13 19:25:41.152162 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:41.152087 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="cc824191ea459b569befd8d139d8d312bd060300c55ac9f104e8ab0fa7a5153b" exitCode=0 Jul 13 19:25:41.152531 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:41.152171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"cc824191ea459b569befd8d139d8d312bd060300c55ac9f104e8ab0fa7a5153b"} Jul 13 19:25:42.594877 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.594842 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:42.595462 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.595021 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 19:25:42.595462 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.595049 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 19:25:42.595462 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.595062 2582 projected.go:196] Error preparing data for projected volume kube-api-access-vndn4 for pod openshift-network-diagnostics/network-check-target-dhhmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:42.595462 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.595121 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4 podName:92e5ae0b-448c-4e5b-bac8-165b749489f0 nodeName:}" failed. No retries permitted until 2026-07-13 19:26:14.595103159 +0000 UTC m=+66.156501195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vndn4" (UniqueName: "kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4") pod "network-check-target-dhhmb" (UID: "92e5ae0b-448c-4e5b-bac8-165b749489f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.695396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.695469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.695552 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.695568 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.695614 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret podName:d918169a-694f-4831-b7f0-9844aa3acecd nodeName:}" failed. No retries permitted until 2026-07-13 19:26:14.695602601 +0000 UTC m=+66.257000623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret") pod "global-pull-secret-syncer-qb8q5" (UID: "d918169a-694f-4831-b7f0-9844aa3acecd") : object "kube-system"/"original-pull-secret" not registered Jul 13 19:25:42.695820 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.695627 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:26:14.695620935 +0000 UTC m=+66.257018957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 19:25:42.955583 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.955553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:42.955740 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.955555 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:42.955740 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.955683 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:25:42.955740 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:42.955693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:42.955904 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.955760 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhhmb" podUID="92e5ae0b-448c-4e5b-bac8-165b749489f0" Jul 13 19:25:42.955904 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:42.955839 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb8q5" podUID="d918169a-694f-4831-b7f0-9844aa3acecd" Jul 13 19:25:44.708412 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.708383 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-83.ec2.internal" event="NodeReady" Jul 13 19:25:44.709004 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.708528 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 19:25:44.742555 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.742525 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:25:44.776147 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.776120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:25:44.776309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.776161 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-v6r8m"] Jul 13 19:25:44.776309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.776239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.779184 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.778958 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 19:25:44.779184 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.779048 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 19:25:44.779390 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.779249 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 19:25:44.779390 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.779250 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jwfmb\"" Jul 13 19:25:44.784443 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.784406 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 19:25:44.812456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.808577 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8g4dk"] Jul 13 19:25:44.812880 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.812856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:44.815850 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.815827 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 13 19:25:44.816307 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.816276 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-887rh\"" Jul 13 19:25:44.816602 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.816586 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 13 19:25:44.834187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.834167 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l4dlq"] Jul 13 19:25:44.834309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.834291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:44.836682 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.836666 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 19:25:44.836780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.836760 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nf5fn\"" Jul 13 19:25:44.837454 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.837421 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 19:25:44.858481 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.858460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-v6r8m"] Jul 13 19:25:44.858574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.858480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:44.858574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.858501 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4dlq"] Jul 13 19:25:44.858574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.858523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8g4dk"] Jul 13 19:25:44.860776 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.860759 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 19:25:44.860864 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.860800 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 19:25:44.861009 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.860991 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9jjnz\"" Jul 13 19:25:44.861081 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.861021 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 19:25:44.915289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:44.915483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915483 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78696\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:44.915788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.915583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:44.930451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.930396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:44.955285 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.955255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:25:44.955463 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.955391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:25:44.955553 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.955255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:25:44.958708 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.958690 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n645d\"" Jul 13 19:25:44.958821 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.958691 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzrsk\"" Jul 13 19:25:44.958978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.958963 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 19:25:44.959066 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.958982 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 19:25:44.959200 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.959187 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 19:25:44.959266 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:44.959214 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 19:25:45.016671 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/824435cf-d217-414d-aaa1-d54901a2dc69-cert\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.016815 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fg8\" (UniqueName: \"kubernetes.io/projected/824435cf-d217-414d-aaa1-d54901a2dc69-kube-api-access-26fg8\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.016815 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.016815 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78696\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.016958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-tmp-dir\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.016958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.016958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.016958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:45.016958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.016927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xks9k\" (UniqueName: \"kubernetes.io/projected/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-kube-api-access-xks9k\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.017191 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.017004 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:25:45.017191 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.017025 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:25:45.017191 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017191 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.017124 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:25:45.517089155 +0000 UTC m=+37.078487196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:25:45.017191 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-config-volume\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.017455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:45.017455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:45.017975 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.017739 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:25:45.017975 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.017784 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:45.517768217 +0000 UTC m=+37.079166239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:25:45.017975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.017975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.017941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.018213 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.018191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.019419 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.019312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:45.022046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.022024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.022164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.022086 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.025288 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.025269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.025841 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.025816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78696\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.117875 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.117844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/824435cf-d217-414d-aaa1-d54901a2dc69-cert\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.118038 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.117882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fg8\" (UniqueName: \"kubernetes.io/projected/824435cf-d217-414d-aaa1-d54901a2dc69-kube-api-access-26fg8\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.118038 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.117904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.118038 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.117969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-tmp-dir\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.118192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.118035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xks9k\" (UniqueName: \"kubernetes.io/projected/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-kube-api-access-xks9k\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.118192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.118080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-config-volume\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.118192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.118164 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:25:45.118327 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.118223 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:25:45.618205363 +0000 UTC m=+37.179603388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:25:45.118327 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.118272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-tmp-dir\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.118561 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.118545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-config-volume\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.120290 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.120269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/824435cf-d217-414d-aaa1-d54901a2dc69-cert\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.129387 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.129368 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fg8\" (UniqueName: \"kubernetes.io/projected/824435cf-d217-414d-aaa1-d54901a2dc69-kube-api-access-26fg8\") pod \"ingress-canary-l4dlq\" (UID: \"824435cf-d217-414d-aaa1-d54901a2dc69\") " pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.129387 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.129381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xks9k\" (UniqueName: \"kubernetes.io/projected/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-kube-api-access-xks9k\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.167888 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.167868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4dlq" Jul 13 19:25:45.327817 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.327611 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4dlq"] Jul 13 19:25:45.378125 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:25:45.378024 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824435cf_d217_414d_aaa1_d54901a2dc69.slice/crio-f9d96c54dfdb41192fd8c5cddc89af2ed8dd8386a703fe1703d4301b841bcd04 WatchSource:0}: Error finding container f9d96c54dfdb41192fd8c5cddc89af2ed8dd8386a703fe1703d4301b841bcd04: Status 404 returned error can't find the container with id f9d96c54dfdb41192fd8c5cddc89af2ed8dd8386a703fe1703d4301b841bcd04 Jul 13 19:25:45.521790 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.521754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:45.521937 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.521882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:45.521937 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.521886 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:25:45.522019 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.521991 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:46.521969402 +0000 UTC m=+38.083367446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:25:45.522019 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.521929 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:25:45.522019 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.522015 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:25:45.522119 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.522063 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:25:46.522045893 +0000 UTC m=+38.083443914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:25:45.622461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:45.622411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:45.622580 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.622561 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:25:45.622633 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:45.622625 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:25:46.622609858 +0000 UTC m=+38.184007879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:25:46.166532 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.166502 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="0f4a1cf57e6a8abe1f6d646a5606259df23038d20b85abaf08210a262b60ce18" exitCode=0 Jul 13 19:25:46.167060 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.166580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"0f4a1cf57e6a8abe1f6d646a5606259df23038d20b85abaf08210a262b60ce18"} Jul 13 19:25:46.167644 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.167619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4dlq" event={"ID":"824435cf-d217-414d-aaa1-d54901a2dc69","Type":"ContainerStarted","Data":"f9d96c54dfdb41192fd8c5cddc89af2ed8dd8386a703fe1703d4301b841bcd04"} Jul 13 19:25:46.528941 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.528854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:46.529097 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.528952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:46.529097 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.528990 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:25:46.529097 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.529070 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:48.52905147 +0000 UTC m=+40.090449508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:25:46.529097 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.529072 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:25:46.529097 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.529086 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:25:46.529356 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.529160 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:25:48.529142148 +0000 UTC m=+40.090540193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:25:46.630131 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:46.630089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:46.630313 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.630258 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:25:46.630375 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:46.630327 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:25:48.63031219 +0000 UTC m=+40.191710211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:25:47.174182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:47.174140 2582 generic.go:356] "Generic (PLEG): container finished" podID="99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb" containerID="dabb9c4217dc6212a293a8ae4e87e76127192e24ca2f76f522108b9fb0049911" exitCode=0 Jul 13 19:25:47.174587 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:47.174197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerDied","Data":"dabb9c4217dc6212a293a8ae4e87e76127192e24ca2f76f522108b9fb0049911"} Jul 13 19:25:48.180547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.180511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" event={"ID":"99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb","Type":"ContainerStarted","Data":"b5243dd000774b20c21dcd1200f2463d78c9230df4722784238ac3c38b790aca"} Jul 13 19:25:48.205133 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.205071 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5dkxv" podStartSLOduration=8.934445895 podStartE2EDuration="39.205049742s" totalImagesPullingTime="30.270603847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:48.204348876 +0000 UTC m=+39.765746918" watchObservedRunningTime="2026-07-13 19:25:48.205049742 +0000 UTC m=+39.766447785" Jul 13 19:25:48.328259 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.328223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:48.386895 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.386853 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:48.546063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.545983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:48.546063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.546044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:48.546221 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.546136 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:25:48.546221 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.546142 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:25:48.546221 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.546159 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:25:48.546221 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.546185 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:25:52.546172565 +0000 UTC m=+44.107570586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:25:48.546338 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.546236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:25:52.546225423 +0000 UTC m=+44.107623444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:25:48.647509 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:48.647478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:48.647645 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.647626 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:25:48.647695 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:48.647690 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:25:52.647675265 +0000 UTC m=+44.209073286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:25:49.183968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:49.183710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4dlq" event={"ID":"824435cf-d217-414d-aaa1-d54901a2dc69","Type":"ContainerStarted","Data":"2486894ffe508a891d296834c78d526d5f3a7efc4ccc403f6f1fe597856b1c6b"} Jul 13 19:25:49.198149 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:49.198101 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l4dlq" podStartSLOduration=1.718821573 podStartE2EDuration="5.198087138s" totalImagesPullingTime="3.479265565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:25:49.197173347 +0000 UTC m=+40.758571381" watchObservedRunningTime="2026-07-13 19:25:49.198087138 +0000 UTC m=+40.759485179" Jul 13 19:25:50.826678 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:50.826636 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:50.906790 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:50.906754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:51.009045 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:51.008983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:51.078321 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:51.078232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:51.832795 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:51.832754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:51.882561 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:51.882527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:52.386416 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:52.386382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:52.488547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:52.488515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" limit=1024 Jul 13 19:25:52.577131 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:52.577102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:25:52.577267 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:52.577167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:25:52.577267 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.577258 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:25:52.577356 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.577274 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:25:52.577356 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.577285 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:25:52.577356 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.577337 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:26:00.577318219 +0000 UTC m=+52.138716260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:25:52.577356 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.577349 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:26:00.577343683 +0000 UTC m=+52.138741704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:25:52.677811 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:25:52.677732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:25:52.677937 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.677841 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:25:52.677937 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:25:52.677892 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:26:00.677879154 +0000 UTC m=+52.239277175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:26:00.637725 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:00.637687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:00.637747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.637882 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.637900 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.637930 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.637991 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:26:16.637974413 +0000 UTC m=+68.199372439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:26:00.638192 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.638005 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:26:16.637998699 +0000 UTC m=+68.199396720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:26:00.738461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:00.738411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:26:00.738614 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.738582 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:26:00.738667 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:00.738656 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:26:16.738637071 +0000 UTC m=+68.300035093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:26:11.162529 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:11.162500 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q55l7" Jul 13 19:26:14.639390 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.639335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:26:14.642472 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.642452 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 19:26:14.652359 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.652338 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 19:26:14.664345 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.664323 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndn4\" (UniqueName: \"kubernetes.io/projected/92e5ae0b-448c-4e5b-bac8-165b749489f0-kube-api-access-vndn4\") pod \"network-check-target-dhhmb\" (UID: \"92e5ae0b-448c-4e5b-bac8-165b749489f0\") " pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:26:14.669395 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.669379 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n645d\"" Jul 13 19:26:14.677354 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.677338 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:26:14.740230 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.740196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:26:14.740370 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.740346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:26:14.743001 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.742982 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 19:26:14.743093 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.743000 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 19:26:14.751843 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:14.751796 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 19:26:14.751905 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:14.751887 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:27:18.751864793 +0000 UTC m=+130.313262833 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : secret "metrics-daemon-secret" not found Jul 13 19:26:14.754096 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.754073 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d918169a-694f-4831-b7f0-9844aa3acecd-original-pull-secret\") pod \"global-pull-secret-syncer-qb8q5\" (UID: \"d918169a-694f-4831-b7f0-9844aa3acecd\") " pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:26:14.794332 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.794307 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dhhmb"] Jul 13 19:26:14.797112 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:26:14.797091 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e5ae0b_448c_4e5b_bac8_165b749489f0.slice/crio-3fc92ff1370eeab442e842f5dc937a1ad0ebae3d612a83207f9436dc93fd9a45 WatchSource:0}: Error finding container 3fc92ff1370eeab442e842f5dc937a1ad0ebae3d612a83207f9436dc93fd9a45: Status 404 returned error can't find the container with id 3fc92ff1370eeab442e842f5dc937a1ad0ebae3d612a83207f9436dc93fd9a45 Jul 13 19:26:14.973722 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:14.973630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb8q5" Jul 13 19:26:15.087894 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:15.087864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qb8q5"] Jul 13 19:26:15.090782 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:26:15.090758 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd918169a_694f_4831_b7f0_9844aa3acecd.slice/crio-ae14e173c3bbe13f5012cc0f764f74cfeb5c27f300f185260f8a6fce806cc544 WatchSource:0}: Error finding container ae14e173c3bbe13f5012cc0f764f74cfeb5c27f300f185260f8a6fce806cc544: Status 404 returned error can't find the container with id ae14e173c3bbe13f5012cc0f764f74cfeb5c27f300f185260f8a6fce806cc544 Jul 13 19:26:15.241603 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:15.241515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qb8q5" event={"ID":"d918169a-694f-4831-b7f0-9844aa3acecd","Type":"ContainerStarted","Data":"ae14e173c3bbe13f5012cc0f764f74cfeb5c27f300f185260f8a6fce806cc544"} Jul 13 19:26:15.242707 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:15.242673 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dhhmb" event={"ID":"92e5ae0b-448c-4e5b-bac8-165b749489f0","Type":"ContainerStarted","Data":"3fc92ff1370eeab442e842f5dc937a1ad0ebae3d612a83207f9436dc93fd9a45"} Jul 13 19:26:16.656034 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:16.655998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:26:16.656509 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:16.656066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:26:16.656598 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.656576 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:26:16.656648 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.656599 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:26:16.656693 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.656666 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:26:48.656649778 +0000 UTC m=+100.218047800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:26:16.656693 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.656468 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:26:16.660191 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.657565 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:26:48.657542383 +0000 UTC m=+100.218940424 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:26:16.757183 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:16.757100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:26:16.757341 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.757223 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:26:16.757341 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:16.757323 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:26:48.757300491 +0000 UTC m=+100.318698514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:26:18.253829 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:18.253792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dhhmb" event={"ID":"92e5ae0b-448c-4e5b-bac8-165b749489f0","Type":"ContainerStarted","Data":"cd2fb16eef9cd0e461c0804c46e6fb6f71e02cd5556ace99c44479eb286b7c7e"} Jul 13 19:26:18.254282 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:18.253953 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:26:18.270398 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:18.270339 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dhhmb" podStartSLOduration=66.279270057 podStartE2EDuration="1m9.270323016s" totalImagesPullingTime="2.991052959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:26:18.270247998 +0000 UTC m=+69.831646043" watchObservedRunningTime="2026-07-13 19:26:18.270323016 +0000 UTC m=+69.831721041" Jul 13 19:26:20.259893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:20.259857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qb8q5" event={"ID":"d918169a-694f-4831-b7f0-9844aa3acecd","Type":"ContainerStarted","Data":"363c7c8dfefcf1d5ad9bca1e9407af09cf025b5c3393ccf5b06be52e4b89a8c6"} Jul 13 19:26:20.276913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:20.276863 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qb8q5" podStartSLOduration=66.114271542 podStartE2EDuration="1m10.276845747s" totalImagesPullingTime="4.162574205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:26:20.276784446 +0000 UTC m=+71.838182490" watchObservedRunningTime="2026-07-13 19:26:20.276845747 +0000 UTC m=+71.838243792" Jul 13 19:26:48.684635 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:48.684500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:26:48.684635 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:48.684571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:26:48.685184 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.684661 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:26:48.685184 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.684729 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:27:52.684712967 +0000 UTC m=+164.246110992 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:26:48.685184 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.684664 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:26:48.685184 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.684765 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:26:48.685184 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.684822 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:27:52.684809706 +0000 UTC m=+164.246207728 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:26:48.785149 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:48.785118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:26:48.785298 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.785266 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:26:48.785352 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:26:48.785341 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:27:52.785326624 +0000 UTC m=+164.346724645 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:26:49.259013 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:26:49.258982 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dhhmb" Jul 13 19:27:18.794665 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:18.794626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:27:18.795158 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:18.794740 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 19:27:18.795158 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:18.794792 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:20.794777796 +0000 UTC m=+252.356175818 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : secret "metrics-daemon-secret" not found Jul 13 19:27:47.819064 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:47.819017 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" Jul 13 19:27:47.826175 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:47.826147 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" podUID="c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63" Jul 13 19:27:47.843374 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:47.843338 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8g4dk" podUID="30a696cb-16cb-4c68-8d82-03dc5c9f5fdb" Jul 13 19:27:47.979228 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:47.979191 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:27:48.458341 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:48.458315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:27:48.458521 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:48.458315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:27:52.732723 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:52.732668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:52.732739 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.732831 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.732855 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bc6d5577-t7988: secret "image-registry-tls" not found Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.732835 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.732909 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert podName:c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:54.7328948 +0000 UTC m=+286.294292822 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-v6r8m" (UID: "c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63") : secret "networking-console-plugin-cert" not found Jul 13 19:27:52.733104 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.732922 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls podName:68072f52-bbc2-4f9f-9456-dec46034fd0e nodeName:}" failed. No retries permitted until 2026-07-13 19:29:54.732916015 +0000 UTC m=+286.294314036 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls") pod "image-registry-64bc6d5577-t7988" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e") : secret "image-registry-tls" not found Jul 13 19:27:52.833412 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:27:52.833367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:27:52.833591 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.833528 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 19:27:52.833629 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:27:52.833607 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls podName:30a696cb-16cb-4c68-8d82-03dc5c9f5fdb nodeName:}" failed. No retries permitted until 2026-07-13 19:29:54.833590093 +0000 UTC m=+286.394988115 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls") pod "dns-default-8g4dk" (UID: "30a696cb-16cb-4c68-8d82-03dc5c9f5fdb") : secret "dns-default-metrics-tls" not found Jul 13 19:28:01.955463 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:28:01.955353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:28:01.955463 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:28:01.955396 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:29:20.848836 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:20.848783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:29:20.849326 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:20.848961 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 19:29:20.849326 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:20.849049 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs podName:3a9254d4-7197-48a0-a784-bb398df08092 nodeName:}" failed. No retries permitted until 2026-07-13 19:31:22.849033228 +0000 UTC m=+374.410431250 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs") pod "network-metrics-daemon-9qx6r" (UID: "3a9254d4-7197-48a0-a784-bb398df08092") : secret "metrics-daemon-secret" not found Jul 13 19:29:32.569058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.568967 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k"] Jul 13 19:29:32.571085 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.571063 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" Jul 13 19:29:32.571489 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.571455 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5"] Jul 13 19:29:32.573406 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.573385 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9"] Jul 13 19:29:32.573592 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.573516 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.574319 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.574298 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5ljwh\"" Jul 13 19:29:32.575168 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.575148 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-g76j9"] Jul 13 19:29:32.575259 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.575250 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.576326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576307 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 19:29:32.576578 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576391 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 19:29:32.576657 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576599 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 19:29:32.576748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xt7jj\"" Jul 13 19:29:32.576853 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576794 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5747877c9b-7jg6t"] Jul 13 19:29:32.576973 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.576944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.580129 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580104 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580363 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580364 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580752 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.580759 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vsdz7\"" Jul 13 19:29:32.581792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.581272 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.582479 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.582084 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 13 19:29:32.582659 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.582638 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 13 19:29:32.583500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.583481 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 19:29:32.583858 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.583838 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 19:29:32.583986 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.583966 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qf9rm\"" Jul 13 19:29:32.585060 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585038 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 19:29:32.585683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585289 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 13 19:29:32.585683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585391 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 13 19:29:32.585683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585400 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 13 19:29:32.585683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585391 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rrm88\"" Jul 13 19:29:32.585683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585671 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 19:29:32.586058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k"] Jul 13 19:29:32.586058 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.585998 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 13 19:29:32.587748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.587706 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9"] Jul 13 19:29:32.591362 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.591177 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 13 19:29:32.592278 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.592255 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-g76j9"] Jul 13 19:29:32.593450 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.593410 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5"] Jul 13 19:29:32.594561 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.594538 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5747877c9b-7jg6t"] Jul 13 19:29:32.623732 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623708 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jlh\" (UniqueName: \"kubernetes.io/projected/2b4e99e2-7ed5-4281-93c5-976864b6035d-kube-api-access-76jlh\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.623834 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-serving-cert\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.623834 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb6j\" (UniqueName: \"kubernetes.io/projected/002ca3fb-ae05-487c-b94a-c30a5234b608-kube-api-access-qgb6j\") pod \"network-check-source-869dcfcdcd-xjq8k\" (UID: \"002ca3fb-ae05-487c-b94a-c30a5234b608\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" Jul 13 19:29:32.623834 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2b4e99e2-7ed5-4281-93c5-976864b6035d-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.623978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623840 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-tmp\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.623978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgtx\" (UniqueName: \"kubernetes.io/projected/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-kube-api-access-xrgtx\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.623978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.623978 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167e0ad5-cd81-4406-b8da-602edba14d4b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.623989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfl9\" (UniqueName: \"kubernetes.io/projected/167e0ad5-cd81-4406-b8da-602edba14d4b-kube-api-access-bmfl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-default-certificate\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624075 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.624117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.624314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-stats-auth\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.624314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ggw\" (UniqueName: \"kubernetes.io/projected/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-kube-api-access-68ggw\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.624314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167e0ad5-cd81-4406-b8da-602edba14d4b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.624314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.624205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-snapshots\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.670975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.670944 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs"] Jul 13 19:29:32.672936 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.672922 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.675064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.675046 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 13 19:29:32.675258 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.675246 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 13 19:29:32.675762 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.675746 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 13 19:29:32.675991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.675978 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c6tlq\"" Jul 13 19:29:32.689039 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.689014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs"] Jul 13 19:29:32.725418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.725418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-default-certificate\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.725541 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-stats-auth\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ggw\" (UniqueName: \"kubernetes.io/projected/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-kube-api-access-68ggw\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.725606 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:33.225585171 +0000 UTC m=+264.786983207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.725638 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167e0ad5-cd81-4406-b8da-602edba14d4b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.725714 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.725699 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:29:33.225683351 +0000 UTC m=+264.787081375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-snapshots\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jlh\" (UniqueName: \"kubernetes.io/projected/2b4e99e2-7ed5-4281-93c5-976864b6035d-kube-api-access-76jlh\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsgr\" (UniqueName: \"kubernetes.io/projected/ae31434a-aca8-4be5-9154-bbab3a666fbe-kube-api-access-9lsgr\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-serving-cert\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb6j\" (UniqueName: \"kubernetes.io/projected/002ca3fb-ae05-487c-b94a-c30a5234b608-kube-api-access-qgb6j\") pod \"network-check-source-869dcfcdcd-xjq8k\" (UID: \"002ca3fb-ae05-487c-b94a-c30a5234b608\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2b4e99e2-7ed5-4281-93c5-976864b6035d-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.725957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-tmp\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgtx\" (UniqueName: \"kubernetes.io/projected/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-kube-api-access-xrgtx\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167e0ad5-cd81-4406-b8da-602edba14d4b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfl9\" (UniqueName: \"kubernetes.io/projected/167e0ad5-cd81-4406-b8da-602edba14d4b-kube-api-access-bmfl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.726214 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726138 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/167e0ad5-cd81-4406-b8da-602edba14d4b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.726349 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:33.226333629 +0000 UTC m=+264.787731863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-snapshots\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726517 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726671 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-tmp\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.726798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2b4e99e2-7ed5-4281-93c5-976864b6035d-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.727062 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.726850 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.728365 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.728346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-serving-cert\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.728807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.728790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/167e0ad5-cd81-4406-b8da-602edba14d4b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.728872 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.728821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-stats-auth\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.729064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.729048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-default-certificate\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.737139 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.737113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ggw\" (UniqueName: \"kubernetes.io/projected/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-kube-api-access-68ggw\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:32.737462 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.737413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb6j\" (UniqueName: \"kubernetes.io/projected/002ca3fb-ae05-487c-b94a-c30a5234b608-kube-api-access-qgb6j\") pod \"network-check-source-869dcfcdcd-xjq8k\" (UID: \"002ca3fb-ae05-487c-b94a-c30a5234b608\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" Jul 13 19:29:32.737900 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.737873 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfl9\" (UniqueName: \"kubernetes.io/projected/167e0ad5-cd81-4406-b8da-602edba14d4b-kube-api-access-bmfl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-sqtk9\" (UID: \"167e0ad5-cd81-4406-b8da-602edba14d4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.738017 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.738001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jlh\" (UniqueName: \"kubernetes.io/projected/2b4e99e2-7ed5-4281-93c5-976864b6035d-kube-api-access-76jlh\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:32.738087 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.738057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgtx\" (UniqueName: \"kubernetes.io/projected/2b46d62f-d1a6-41a7-ab51-0064bb54a1bb-kube-api-access-xrgtx\") pod \"insights-operator-68d9dd94cf-g76j9\" (UID: \"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb\") " pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:32.826673 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.826581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.826673 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.826649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsgr\" (UniqueName: \"kubernetes.io/projected/ae31434a-aca8-4be5-9154-bbab3a666fbe-kube-api-access-9lsgr\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.826879 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.826734 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 19:29:32.826879 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:32.826818 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls podName:ae31434a-aca8-4be5-9154-bbab3a666fbe nodeName:}" failed. No retries permitted until 2026-07-13 19:29:33.326793454 +0000 UTC m=+264.888191476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls") pod "cluster-samples-operator-65db858894-whcbs" (UID: "ae31434a-aca8-4be5-9154-bbab3a666fbe") : secret "samples-operator-tls" not found Jul 13 19:29:32.835391 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.835356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsgr\" (UniqueName: \"kubernetes.io/projected/ae31434a-aca8-4be5-9154-bbab3a666fbe-kube-api-access-9lsgr\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:32.888342 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.888297 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" Jul 13 19:29:32.902374 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.902350 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" Jul 13 19:29:32.908314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:32.908292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" Jul 13 19:29:33.028288 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.028249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k"] Jul 13 19:29:33.036842 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:33.035970 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002ca3fb_ae05_487c_b94a_c30a5234b608.slice/crio-8ddbe8e72ec1fb3779672f9f8c62f2e9a8f71069a1db73c6b8ab12d85c8b00d2 WatchSource:0}: Error finding container 8ddbe8e72ec1fb3779672f9f8c62f2e9a8f71069a1db73c6b8ab12d85c8b00d2: Status 404 returned error can't find the container with id 8ddbe8e72ec1fb3779672f9f8c62f2e9a8f71069a1db73c6b8ab12d85c8b00d2 Jul 13 19:29:33.046243 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.046218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9"] Jul 13 19:29:33.048652 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:33.048623 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167e0ad5_cd81_4406_b8da_602edba14d4b.slice/crio-420566d2cd2d9aa43b06484ac23d75f1a626a020bfd9373cfce42906ae3ff5b3 WatchSource:0}: Error finding container 420566d2cd2d9aa43b06484ac23d75f1a626a020bfd9373cfce42906ae3ff5b3: Status 404 returned error can't find the container with id 420566d2cd2d9aa43b06484ac23d75f1a626a020bfd9373cfce42906ae3ff5b3 Jul 13 19:29:33.063838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.063818 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-g76j9"] Jul 13 19:29:33.066078 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:33.065996 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46d62f_d1a6_41a7_ab51_0064bb54a1bb.slice/crio-a0dfab58d32c0275d67297dcb44d5dcb39dac738d2bba0e3ab879e7df94a0273 WatchSource:0}: Error finding container a0dfab58d32c0275d67297dcb44d5dcb39dac738d2bba0e3ab879e7df94a0273: Status 404 returned error can't find the container with id a0dfab58d32c0275d67297dcb44d5dcb39dac738d2bba0e3ab879e7df94a0273 Jul 13 19:29:33.230209 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.230172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.230221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.230244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.230345 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.230353 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.230372 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:34.230351682 +0000 UTC m=+265.791749719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.230396 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:29:34.230386897 +0000 UTC m=+265.791784925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:33.230410 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.230413 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:34.230403088 +0000 UTC m=+265.791801118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:33.331222 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.331185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:33.331347 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.331324 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 19:29:33.331448 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:33.331389 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls podName:ae31434a-aca8-4be5-9154-bbab3a666fbe nodeName:}" failed. No retries permitted until 2026-07-13 19:29:34.33137385 +0000 UTC m=+265.892771871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls") pod "cluster-samples-operator-65db858894-whcbs" (UID: "ae31434a-aca8-4be5-9154-bbab3a666fbe") : secret "samples-operator-tls" not found Jul 13 19:29:33.697607 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.697569 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" event={"ID":"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb","Type":"ContainerStarted","Data":"a0dfab58d32c0275d67297dcb44d5dcb39dac738d2bba0e3ab879e7df94a0273"} Jul 13 19:29:33.698700 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.698674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" event={"ID":"167e0ad5-cd81-4406-b8da-602edba14d4b","Type":"ContainerStarted","Data":"420566d2cd2d9aa43b06484ac23d75f1a626a020bfd9373cfce42906ae3ff5b3"} Jul 13 19:29:33.704582 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.704553 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" event={"ID":"002ca3fb-ae05-487c-b94a-c30a5234b608","Type":"ContainerStarted","Data":"67efe043264edf43d079e4b5afede403b2a791fce5da11c2d311be3f16c19d10"} Jul 13 19:29:33.704683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.704627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" event={"ID":"002ca3fb-ae05-487c-b94a-c30a5234b608","Type":"ContainerStarted","Data":"8ddbe8e72ec1fb3779672f9f8c62f2e9a8f71069a1db73c6b8ab12d85c8b00d2"} Jul 13 19:29:33.721626 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:33.721574 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-xjq8k" podStartSLOduration=1.721555468 podStartE2EDuration="1.721555468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:33.721176347 +0000 UTC m=+265.282574392" watchObservedRunningTime="2026-07-13 19:29:33.721555468 +0000 UTC m=+265.282953513" Jul 13 19:29:34.239952 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:34.239911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:34.239981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:34.240018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.240079 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.240108 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:36.240087778 +0000 UTC m=+267.801485803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.240114 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.240136 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:36.240124031 +0000 UTC m=+267.801522076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:34.240293 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.240185 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:29:36.240169293 +0000 UTC m=+267.801567321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:34.340555 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:34.340517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:34.340742 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.340687 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 19:29:34.340805 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:34.340760 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls podName:ae31434a-aca8-4be5-9154-bbab3a666fbe nodeName:}" failed. No retries permitted until 2026-07-13 19:29:36.340741424 +0000 UTC m=+267.902139452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls") pod "cluster-samples-operator-65db858894-whcbs" (UID: "ae31434a-aca8-4be5-9154-bbab3a666fbe") : secret "samples-operator-tls" not found Jul 13 19:29:35.712684 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:35.712643 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" event={"ID":"167e0ad5-cd81-4406-b8da-602edba14d4b","Type":"ContainerStarted","Data":"6fa53572f43e5e53eca65a5b9113e8c35cc10f2ac2ccd406e5a976237dd606f4"} Jul 13 19:29:35.728683 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:35.728638 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" podStartSLOduration=1.572202346 podStartE2EDuration="3.728622857s" totalImagesPullingTime="2.156420511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:35.727691288 +0000 UTC m=+267.289089333" watchObservedRunningTime="2026-07-13 19:29:35.728622857 +0000 UTC m=+267.290020900" Jul 13 19:29:36.257641 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:36.257602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:36.257836 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:36.257661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:36.257836 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:36.257694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:36.257836 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.257767 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.257750561 +0000 UTC m=+271.819148589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:36.257836 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.257825 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:36.257836 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.257825 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:36.258032 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.257875 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.257862922 +0000 UTC m=+271.819260944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:36.258032 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.257890 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.257883549 +0000 UTC m=+271.819281571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:36.358688 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:36.358653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:36.358858 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.358818 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 19:29:36.358905 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:36.358886 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls podName:ae31434a-aca8-4be5-9154-bbab3a666fbe nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.358870654 +0000 UTC m=+271.920268676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls") pod "cluster-samples-operator-65db858894-whcbs" (UID: "ae31434a-aca8-4be5-9154-bbab3a666fbe") : secret "samples-operator-tls" not found Jul 13 19:29:37.720910 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:37.720876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" event={"ID":"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb","Type":"ContainerStarted","Data":"e2b653cb9239361ba040917f10f9f134ca615aa4b6d7008a59bb0137b5acecaa"} Jul 13 19:29:37.743533 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:37.743489 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" podStartSLOduration=1.483032246 podStartE2EDuration="5.743474812s" totalImagesPullingTime="4.260442566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:37.742756353 +0000 UTC m=+269.304154398" watchObservedRunningTime="2026-07-13 19:29:37.743474812 +0000 UTC m=+269.304872856" Jul 13 19:29:38.386712 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.386673 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bhvqg"] Jul 13 19:29:38.389686 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.389644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.392795 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.392768 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 19:29:38.392963 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.392813 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 19:29:38.392963 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.392850 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwnk\"" Jul 13 19:29:38.403588 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.403560 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhvqg"] Jul 13 19:29:38.477278 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.477245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.477312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.477346 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b59g\" (UniqueName: \"kubernetes.io/projected/e06a0954-27f7-4273-aa18-63542590dce6-kube-api-access-5b59g\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.477366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e06a0954-27f7-4273-aa18-63542590dce6-crio-socket\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.477574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.477555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e06a0954-27f7-4273-aa18-63542590dce6-data-volume\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578032 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.577996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e06a0954-27f7-4273-aa18-63542590dce6-crio-socket\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e06a0954-27f7-4273-aa18-63542590dce6-data-volume\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e06a0954-27f7-4273-aa18-63542590dce6-crio-socket\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578155 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b59g\" (UniqueName: \"kubernetes.io/projected/e06a0954-27f7-4273-aa18-63542590dce6-kube-api-access-5b59g\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.578395 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:38.578238 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:38.578395 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:38.578297 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:39.078281368 +0000 UTC m=+270.639679393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:38.578395 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:38.578315 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 19:29:38.578395 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:38.578377 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:39.078361732 +0000 UTC m=+270.639759753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : configmap "kube-rbac-proxy" not found Jul 13 19:29:38.578620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.578464 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e06a0954-27f7-4273-aa18-63542590dce6-data-volume\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:38.588411 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:38.588391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b59g\" (UniqueName: \"kubernetes.io/projected/e06a0954-27f7-4273-aa18-63542590dce6-kube-api-access-5b59g\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:39.082093 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:39.082058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:39.082467 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:39.082104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:39.082467 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:39.082195 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 19:29:39.082467 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:39.082199 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:39.082467 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:39.082255 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.082241538 +0000 UTC m=+271.643639560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : configmap "kube-rbac-proxy" not found Jul 13 19:29:39.082467 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:39.082267 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:40.082261594 +0000 UTC m=+271.643659616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:40.091034 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.091002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:40.091408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.091048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:40.091408 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.091120 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 19:29:40.091408 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.091148 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:40.091408 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.091171 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:42.091157166 +0000 UTC m=+273.652555189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : configmap "kube-rbac-proxy" not found Jul 13 19:29:40.091408 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.091204 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:42.091190511 +0000 UTC m=+273.652588532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:40.293048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.293010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:40.293197 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.293061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:40.293197 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.293088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:40.293197 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.293168 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:40.293197 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.293171 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:40.293197 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.293186 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:48.293165037 +0000 UTC m=+279.854563059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:40.293374 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.293209 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:29:48.2931976 +0000 UTC m=+279.854595622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:40.293374 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.293222 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:48.29321563 +0000 UTC m=+279.854613652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:40.393414 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.393378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:40.393610 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.393527 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 19:29:40.393610 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:40.393589 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls podName:ae31434a-aca8-4be5-9154-bbab3a666fbe nodeName:}" failed. No retries permitted until 2026-07-13 19:29:48.393574165 +0000 UTC m=+279.954972186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls") pod "cluster-samples-operator-65db858894-whcbs" (UID: "ae31434a-aca8-4be5-9154-bbab3a666fbe") : secret "samples-operator-tls" not found Jul 13 19:29:40.415536 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.415498 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xfm2j"] Jul 13 19:29:40.417607 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.417591 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.419998 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.419975 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 19:29:40.420086 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.420017 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 19:29:40.420866 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.420853 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 19:29:40.420929 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.420906 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 19:29:40.420929 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.420908 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c8nj7\"" Jul 13 19:29:40.427476 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.427452 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xfm2j"] Jul 13 19:29:40.494305 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.494270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a94349-3ff3-4217-b324-53f5f337e72f-signing-cabundle\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.494493 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.494375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88drj\" (UniqueName: \"kubernetes.io/projected/99a94349-3ff3-4217-b324-53f5f337e72f-kube-api-access-88drj\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.494493 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.494416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a94349-3ff3-4217-b324-53f5f337e72f-signing-key\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.595772 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.595731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a94349-3ff3-4217-b324-53f5f337e72f-signing-key\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.595973 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.595812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a94349-3ff3-4217-b324-53f5f337e72f-signing-cabundle\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.595973 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.595876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88drj\" (UniqueName: \"kubernetes.io/projected/99a94349-3ff3-4217-b324-53f5f337e72f-kube-api-access-88drj\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.596524 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.596500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99a94349-3ff3-4217-b324-53f5f337e72f-signing-cabundle\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.598228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.598200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99a94349-3ff3-4217-b324-53f5f337e72f-signing-key\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.606275 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.606250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88drj\" (UniqueName: \"kubernetes.io/projected/99a94349-3ff3-4217-b324-53f5f337e72f-kube-api-access-88drj\") pod \"service-ca-79f68b9f4f-xfm2j\" (UID: \"99a94349-3ff3-4217-b324-53f5f337e72f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.727035 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.726948 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" Jul 13 19:29:40.849137 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:40.849105 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xfm2j"] Jul 13 19:29:40.851621 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:40.851592 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a94349_3ff3_4217_b324_53f5f337e72f.slice/crio-09948aea58da88b02a15524723ba73387644c26682211f22da5191aa9a6738aa WatchSource:0}: Error finding container 09948aea58da88b02a15524723ba73387644c26682211f22da5191aa9a6738aa: Status 404 returned error can't find the container with id 09948aea58da88b02a15524723ba73387644c26682211f22da5191aa9a6738aa Jul 13 19:29:41.177105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:41.177079 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t762r_d6cbfa7b-786e-425b-82e5-10d3faaf9790/dns-node-resolver/0.log" Jul 13 19:29:41.734292 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:41.734254 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" event={"ID":"99a94349-3ff3-4217-b324-53f5f337e72f","Type":"ContainerStarted","Data":"09948aea58da88b02a15524723ba73387644c26682211f22da5191aa9a6738aa"} Jul 13 19:29:42.109507 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:42.109475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:42.109656 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:42.109534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:42.109656 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:42.109623 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 19:29:42.109656 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:42.109629 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:42.109756 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:42.109686 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:46.109667357 +0000 UTC m=+277.671065382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:42.109756 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:42.109701 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:46.109693673 +0000 UTC m=+277.671091699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : configmap "kube-rbac-proxy" not found Jul 13 19:29:42.176014 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:42.175991 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccvgs_33ea8156-02d6-4487-8148-d6c4c28fa8e0/node-ca/0.log" Jul 13 19:29:42.978224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:42.978191 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4dlq_824435cf-d217-414d-aaa1-d54901a2dc69/serve-healthcheck-canary/0.log" Jul 13 19:29:43.742254 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:43.742220 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" event={"ID":"99a94349-3ff3-4217-b324-53f5f337e72f","Type":"ContainerStarted","Data":"1cd8729887e5f1e6c4623fee6fc468e6839178f47a5415359fc5668c79732289"} Jul 13 19:29:43.757260 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:43.757212 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-xfm2j" podStartSLOduration=1.739530372 podStartE2EDuration="3.757193889s" totalImagesPullingTime="2.017663517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:43.75701463 +0000 UTC m=+275.318412673" watchObservedRunningTime="2026-07-13 19:29:43.757193889 +0000 UTC m=+275.318591934" Jul 13 19:29:43.780059 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:43.780030 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-sqtk9_167e0ad5-cd81-4406-b8da-602edba14d4b/kube-storage-version-migrator-operator/0.log" Jul 13 19:29:46.146345 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:46.146308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:46.146753 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:46.146442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:46.146753 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:46.146455 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 19:29:46.146753 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:46.146527 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:54.146509341 +0000 UTC m=+285.707907362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : configmap "kube-rbac-proxy" not found Jul 13 19:29:46.146753 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:46.146538 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:46.146753 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:46.146576 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:29:54.146565295 +0000 UTC m=+285.707963317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:48.365779 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.365738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.365807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.365837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:48.365920 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:48.365938 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:48.365965 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:30:04.365942521 +0000 UTC m=+295.927340548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : configmap references non-existent config key: service-ca.crt Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:48.366000 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs podName:dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2 nodeName:}" failed. No retries permitted until 2026-07-13 19:30:04.365981705 +0000 UTC m=+295.927379732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs") pod "router-default-5747877c9b-7jg6t" (UID: "dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2") : secret "router-metrics-certs-default" not found Jul 13 19:29:48.366217 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:48.366019 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls podName:2b4e99e2-7ed5-4281-93c5-976864b6035d nodeName:}" failed. No retries permitted until 2026-07-13 19:30:04.366009178 +0000 UTC m=+295.927407204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-6m4h5" (UID: "2b4e99e2-7ed5-4281-93c5-976864b6035d") : secret "cluster-monitoring-operator-tls" not found Jul 13 19:29:48.466320 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.466281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:48.468969 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.468937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae31434a-aca8-4be5-9154-bbab3a666fbe-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-whcbs\" (UID: \"ae31434a-aca8-4be5-9154-bbab3a666fbe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:48.581630 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.581577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" Jul 13 19:29:48.705010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.704980 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs"] Jul 13 19:29:48.758134 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:48.758106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" event={"ID":"ae31434a-aca8-4be5-9154-bbab3a666fbe","Type":"ContainerStarted","Data":"0bb384c5c7124e69dd3f5151de4b9ed50def2f9a31f4f90494de815e1233fdb9"} Jul 13 19:29:50.766967 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:50.766934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" event={"ID":"ae31434a-aca8-4be5-9154-bbab3a666fbe","Type":"ContainerStarted","Data":"f43d735898e7eb742b3acbbce300a1e6260dead2c769ff634b2d4899fc746c98"} Jul 13 19:29:50.767311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:50.766991 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" event={"ID":"ae31434a-aca8-4be5-9154-bbab3a666fbe","Type":"ContainerStarted","Data":"0120976e6d4d560281ce68fa2ca53108dd85cb18afec919c925180b50edb1add"} Jul 13 19:29:50.784538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:50.784489 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-whcbs" podStartSLOduration=16.942096172 podStartE2EDuration="18.784473409s" totalImagesPullingTime="1.842377237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:50.783987433 +0000 UTC m=+282.345385476" watchObservedRunningTime="2026-07-13 19:29:50.784473409 +0000 UTC m=+282.345871453" Jul 13 19:29:51.459471 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:51.459407 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" podUID="c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63" Jul 13 19:29:51.459471 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:51.459407 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8g4dk" podUID="30a696cb-16cb-4c68-8d82-03dc5c9f5fdb" Jul 13 19:29:51.769745 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:51.769663 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:29:51.769745 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:51.769662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:29:54.215315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.215268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:54.215754 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.215336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:54.215754 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:54.215464 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 19:29:54.215754 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:29:54.215547 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls podName:e06a0954-27f7-4273-aa18-63542590dce6 nodeName:}" failed. No retries permitted until 2026-07-13 19:30:10.215524984 +0000 UTC m=+301.776923006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhvqg" (UID: "e06a0954-27f7-4273-aa18-63542590dce6") : secret "insights-runtime-extractor-tls" not found Jul 13 19:29:54.215998 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.215977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e06a0954-27f7-4273-aa18-63542590dce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:29:54.820403 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.820362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:29:54.820606 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.820413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:29:54.823053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.823023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-v6r8m\" (UID: \"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:29:54.823053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.823045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"image-registry-64bc6d5577-t7988\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:29:54.921449 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.921379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:29:54.923890 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:54.923857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30a696cb-16cb-4c68-8d82-03dc5c9f5fdb-metrics-tls\") pod \"dns-default-8g4dk\" (UID: \"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb\") " pod="openshift-dns/dns-default-8g4dk" Jul 13 19:29:55.059192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.059156 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jwfmb\"" Jul 13 19:29:55.067197 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.067169 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:29:55.073056 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.072994 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nf5fn\"" Jul 13 19:29:55.073056 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.073001 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-887rh\"" Jul 13 19:29:55.080951 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.080924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" Jul 13 19:29:55.080951 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.080943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:29:55.225124 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.224918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:29:55.228158 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:55.228131 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68072f52_bbc2_4f9f_9456_dec46034fd0e.slice/crio-18bbb513b9bc608ce91068a134c15c310e6accd5726eee2075e8712c6463557a WatchSource:0}: Error finding container 18bbb513b9bc608ce91068a134c15c310e6accd5726eee2075e8712c6463557a: Status 404 returned error can't find the container with id 18bbb513b9bc608ce91068a134c15c310e6accd5726eee2075e8712c6463557a Jul 13 19:29:55.231254 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.231218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8g4dk"] Jul 13 19:29:55.233927 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:55.233896 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a696cb_16cb_4c68_8d82_03dc5c9f5fdb.slice/crio-1d492cd5452b324859b3c5f7958816180c6d3df76c4481c6f63eab49d7fc9e59 WatchSource:0}: Error finding container 1d492cd5452b324859b3c5f7958816180c6d3df76c4481c6f63eab49d7fc9e59: Status 404 returned error can't find the container with id 1d492cd5452b324859b3c5f7958816180c6d3df76c4481c6f63eab49d7fc9e59 Jul 13 19:29:55.242748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.242726 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-v6r8m"] Jul 13 19:29:55.244864 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:29:55.244839 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc959e1e6_d8a9_4cfc_8a10_f8aa63f27e63.slice/crio-dbc942e2a76c6eb8a3dfadddbd3fe21692aaf86bcb9870d2bedc92eb6c5d63db WatchSource:0}: Error finding container dbc942e2a76c6eb8a3dfadddbd3fe21692aaf86bcb9870d2bedc92eb6c5d63db: Status 404 returned error can't find the container with id dbc942e2a76c6eb8a3dfadddbd3fe21692aaf86bcb9870d2bedc92eb6c5d63db Jul 13 19:29:55.783594 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.783556 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" event={"ID":"68072f52-bbc2-4f9f-9456-dec46034fd0e","Type":"ContainerStarted","Data":"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d"} Jul 13 19:29:55.783791 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.783615 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:29:55.783791 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.783637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" event={"ID":"68072f52-bbc2-4f9f-9456-dec46034fd0e","Type":"ContainerStarted","Data":"18bbb513b9bc608ce91068a134c15c310e6accd5726eee2075e8712c6463557a"} Jul 13 19:29:55.784690 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.784668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8g4dk" event={"ID":"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb","Type":"ContainerStarted","Data":"1d492cd5452b324859b3c5f7958816180c6d3df76c4481c6f63eab49d7fc9e59"} Jul 13 19:29:55.785731 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.785710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" event={"ID":"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63","Type":"ContainerStarted","Data":"dbc942e2a76c6eb8a3dfadddbd3fe21692aaf86bcb9870d2bedc92eb6c5d63db"} Jul 13 19:29:55.802705 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:55.802659 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" podStartSLOduration=285.802644083 podStartE2EDuration="4m45.802644083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:55.80157486 +0000 UTC m=+287.362972921" watchObservedRunningTime="2026-07-13 19:29:55.802644083 +0000 UTC m=+287.364042126" Jul 13 19:29:57.794000 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:57.793967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8g4dk" event={"ID":"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb","Type":"ContainerStarted","Data":"b7ff044cf3e064c2ca47c60081e82af6cfc8427b2ce605f936ccd2a61af875e6"} Jul 13 19:29:57.794446 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:57.794068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8g4dk" event={"ID":"30a696cb-16cb-4c68-8d82-03dc5c9f5fdb","Type":"ContainerStarted","Data":"b4ab03bd3ce7f46bcd432c62a3eb307ddb859764c47a8c11839ca6ef6f69e82e"} Jul 13 19:29:57.794446 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:57.794115 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:29:57.795555 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:57.795531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" event={"ID":"c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63","Type":"ContainerStarted","Data":"8e41900bdb0954eb636e4fa3375da2f5463056347b7e42e123e5cbd94d7d3c48"} Jul 13 19:29:57.809981 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:29:57.809925 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8g4dk" podStartSLOduration=251.425970124 podStartE2EDuration="4m13.809905705s" totalImagesPullingTime="2.383935581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:57.808586717 +0000 UTC m=+289.369984789" watchObservedRunningTime="2026-07-13 19:29:57.809905705 +0000 UTC m=+289.371303750" Jul 13 19:30:04.398614 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.398552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:04.398614 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.398623 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:04.399245 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.398660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:30:04.399245 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.399143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-service-ca-bundle\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:04.401097 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.401077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2-metrics-certs\") pod \"router-default-5747877c9b-7jg6t\" (UID: \"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2\") " pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:04.401158 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.401102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4e99e2-7ed5-4281-93c5-976864b6035d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-6m4h5\" (UID: \"2b4e99e2-7ed5-4281-93c5-976864b6035d\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:30:04.413023 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.412999 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:04.536967 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.536906 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-v6r8m" podStartSLOduration=276.166970168 podStartE2EDuration="4m38.536889361s" totalImagesPullingTime="2.369919193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:57.822930669 +0000 UTC m=+289.384328733" watchObservedRunningTime="2026-07-13 19:30:04.536889361 +0000 UTC m=+296.098287405" Jul 13 19:30:04.538302 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.538281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5747877c9b-7jg6t"] Jul 13 19:30:04.540113 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:04.540082 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb8a5e6_05d7_4953_ab1d_709e18f9b4b2.slice/crio-2b19a5316c3e760a6d319848ebb6d5d78ad2681fe8de1800139fbfaa84b7001e WatchSource:0}: Error finding container 2b19a5316c3e760a6d319848ebb6d5d78ad2681fe8de1800139fbfaa84b7001e: Status 404 returned error can't find the container with id 2b19a5316c3e760a6d319848ebb6d5d78ad2681fe8de1800139fbfaa84b7001e Jul 13 19:30:04.696251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.696161 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" Jul 13 19:30:04.817808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.817771 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5747877c9b-7jg6t" event={"ID":"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2","Type":"ContainerStarted","Data":"8adcc757c92ad66219eaa57e3c252149df580b2bc97908197429fe2119288502"} Jul 13 19:30:04.817982 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.817828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5747877c9b-7jg6t" event={"ID":"dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2","Type":"ContainerStarted","Data":"2b19a5316c3e760a6d319848ebb6d5d78ad2681fe8de1800139fbfaa84b7001e"} Jul 13 19:30:04.822019 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.821995 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5"] Jul 13 19:30:04.825208 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:04.825181 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4e99e2_7ed5_4281_93c5_976864b6035d.slice/crio-ae33674ced7850091a8aa98f11b39f3166ed3db87db3d6c0d5869c316728498b WatchSource:0}: Error finding container ae33674ced7850091a8aa98f11b39f3166ed3db87db3d6c0d5869c316728498b: Status 404 returned error can't find the container with id ae33674ced7850091a8aa98f11b39f3166ed3db87db3d6c0d5869c316728498b Jul 13 19:30:04.836044 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:04.835998 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5747877c9b-7jg6t" podStartSLOduration=32.835983998 podStartE2EDuration="32.835983998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:04.835218868 +0000 UTC m=+296.396616912" watchObservedRunningTime="2026-07-13 19:30:04.835983998 +0000 UTC m=+296.397382035" Jul 13 19:30:04.956487 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:30:04.956378 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9qx6r" podUID="3a9254d4-7197-48a0-a784-bb398df08092" Jul 13 19:30:05.261185 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.261104 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:05.263971 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.263951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.266257 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266239 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 19:30:05.266544 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266520 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 19:30:05.266640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266559 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dlpwn\"" Jul 13 19:30:05.266640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266565 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 19:30:05.266640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266525 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 19:30:05.266640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.266599 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 19:30:05.267251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.267235 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 19:30:05.267326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.267272 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 19:30:05.274309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.274281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:05.407758 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.407721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.408224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.407783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.408224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.407935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.408224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.407981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.408224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.408069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.408224 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.408103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gn7\" (UniqueName: \"kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.413708 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.413679 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:05.416121 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.416103 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:05.509162 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509162 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509421 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509505 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509505 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509606 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509515 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gn7\" (UniqueName: \"kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.509993 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.509968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.510084 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.510007 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.510352 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.510329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.511845 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.511789 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.511944 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.511910 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.517995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.517975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gn7\" (UniqueName: \"kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7\") pod \"console-7c6c44ff65-rbqrl\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.572907 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.572865 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:05.696478 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.696300 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:05.698802 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:05.698770 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa83aacd_b1b8_4c16_bd76_a9945f173ddf.slice/crio-5f9fc23e0626b974e35ea1b681c9c4ae0d650db0c7efbae6e59e518ddbc46a4a WatchSource:0}: Error finding container 5f9fc23e0626b974e35ea1b681c9c4ae0d650db0c7efbae6e59e518ddbc46a4a: Status 404 returned error can't find the container with id 5f9fc23e0626b974e35ea1b681c9c4ae0d650db0c7efbae6e59e518ddbc46a4a Jul 13 19:30:05.824168 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.824073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c44ff65-rbqrl" event={"ID":"aa83aacd-b1b8-4c16-bd76-a9945f173ddf","Type":"ContainerStarted","Data":"5f9fc23e0626b974e35ea1b681c9c4ae0d650db0c7efbae6e59e518ddbc46a4a"} Jul 13 19:30:05.825246 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.825223 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" event={"ID":"2b4e99e2-7ed5-4281-93c5-976864b6035d","Type":"ContainerStarted","Data":"ae33674ced7850091a8aa98f11b39f3166ed3db87db3d6c0d5869c316728498b"} Jul 13 19:30:05.825437 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.825410 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:05.826586 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:05.826567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5747877c9b-7jg6t" Jul 13 19:30:07.801922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:07.801838 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8g4dk" Jul 13 19:30:07.844098 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:07.844026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" event={"ID":"2b4e99e2-7ed5-4281-93c5-976864b6035d","Type":"ContainerStarted","Data":"da090f468eec90ecb3b7dedfd764ba6a4d3e747a465531b53327a90a442a3757"} Jul 13 19:30:07.863904 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:07.862562 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-6m4h5" podStartSLOduration=33.145386005 podStartE2EDuration="35.862543832s" totalImagesPullingTime="2.717157827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:07.862290155 +0000 UTC m=+299.423688411" watchObservedRunningTime="2026-07-13 19:30:07.862543832 +0000 UTC m=+299.423941875" Jul 13 19:30:08.838554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:08.838522 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:30:08.838995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:08.838712 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:30:08.841181 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:08.841157 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 19:30:09.851749 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:09.851715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c44ff65-rbqrl" event={"ID":"aa83aacd-b1b8-4c16-bd76-a9945f173ddf","Type":"ContainerStarted","Data":"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e"} Jul 13 19:30:09.870441 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:09.870359 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c6c44ff65-rbqrl" podStartSLOduration=1.720974483 podStartE2EDuration="4.870341804s" totalImagesPullingTime="3.149367321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:09.869406744 +0000 UTC m=+301.430804804" watchObservedRunningTime="2026-07-13 19:30:09.870341804 +0000 UTC m=+301.431739847" Jul 13 19:30:10.250754 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.250671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:30:10.253115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.253084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e06a0954-27f7-4273-aa18-63542590dce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhvqg\" (UID: \"e06a0954-27f7-4273-aa18-63542590dce6\") " pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:30:10.502811 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.502725 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwnk\"" Jul 13 19:30:10.510843 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.510820 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhvqg" Jul 13 19:30:10.683174 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.683147 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhvqg"] Jul 13 19:30:10.685203 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:10.685170 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06a0954_27f7_4273_aa18_63542590dce6.slice/crio-ad19975567694dfbc42784defe1887f1e1a82e3c2adcf3af24aa2e7e95db29c0 WatchSource:0}: Error finding container ad19975567694dfbc42784defe1887f1e1a82e3c2adcf3af24aa2e7e95db29c0: Status 404 returned error can't find the container with id ad19975567694dfbc42784defe1887f1e1a82e3c2adcf3af24aa2e7e95db29c0 Jul 13 19:30:10.686590 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.686567 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 19:30:10.856681 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:10.856644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhvqg" event={"ID":"e06a0954-27f7-4273-aa18-63542590dce6","Type":"ContainerStarted","Data":"ad19975567694dfbc42784defe1887f1e1a82e3c2adcf3af24aa2e7e95db29c0"} Jul 13 19:30:13.041833 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.040492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:30:13.043064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.043034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.053572 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.053545 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 19:30:13.054462 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.054399 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrjg\" (UniqueName: \"kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072567 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.072488 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.072592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrjg\" (UniqueName: \"kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.173636 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.174064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.174064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.173914 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.174635 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.174244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.175162 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.174835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.175408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.175385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.178882 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.178842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.180783 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.180739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.185252 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.185211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrjg\" (UniqueName: \"kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg\") pod \"console-76fc887548-wqgm7\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.365200 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.365163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:13.516111 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.516073 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:30:13.520866 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:13.520835 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c5c561_83d2_4804_bc8b_20f17ba1d0dd.slice/crio-8d74be408cca81974fc1810f51e6e3fdd52a3bd8b4f7336301467ba8943147ab WatchSource:0}: Error finding container 8d74be408cca81974fc1810f51e6e3fdd52a3bd8b4f7336301467ba8943147ab: Status 404 returned error can't find the container with id 8d74be408cca81974fc1810f51e6e3fdd52a3bd8b4f7336301467ba8943147ab Jul 13 19:30:13.869772 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.869733 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc887548-wqgm7" event={"ID":"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd","Type":"ContainerStarted","Data":"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633"} Jul 13 19:30:13.869942 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.869796 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc887548-wqgm7" event={"ID":"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd","Type":"ContainerStarted","Data":"8d74be408cca81974fc1810f51e6e3fdd52a3bd8b4f7336301467ba8943147ab"} Jul 13 19:30:13.889674 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:13.889617 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76fc887548-wqgm7" podStartSLOduration=0.88959478 podStartE2EDuration="889.59478ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:13.88812107 +0000 UTC m=+305.449519117" watchObservedRunningTime="2026-07-13 19:30:13.88959478 +0000 UTC m=+305.450992824" Jul 13 19:30:15.072703 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.072654 2582 patch_prober.go:28] interesting pod/image-registry-64bc6d5577-t7988 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 13 19:30:15.073202 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.072711 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 19:30:15.573098 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.573055 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:15.573291 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.573126 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:15.578508 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.578482 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:15.887748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:15.887722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:16.794003 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:16.793975 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:30:16.887589 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:16.887553 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhvqg" event={"ID":"e06a0954-27f7-4273-aa18-63542590dce6","Type":"ContainerStarted","Data":"bef1ec1229c3e9fcf151499aad3b31366f7836f58c4c9dfed515e3518af99441"} Jul 13 19:30:17.954700 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:17.954668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:30:18.898052 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:18.898014 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhvqg" event={"ID":"e06a0954-27f7-4273-aa18-63542590dce6","Type":"ContainerStarted","Data":"39d4143bb956c30364af3ab086ab2f66614c8efdd2d13680e32dbcff511fe0ed"} Jul 13 19:30:20.906910 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:20.906877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhvqg" event={"ID":"e06a0954-27f7-4273-aa18-63542590dce6","Type":"ContainerStarted","Data":"f347cb56984a31a51720566425f345ca0786126f689e90316cc1b58fb433e09a"} Jul 13 19:30:20.925691 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:20.925635 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bhvqg" podStartSLOduration=33.188533074 podStartE2EDuration="42.925621144s" totalImagesPullingTime="9.73708807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:29:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:20.923748178 +0000 UTC m=+312.485146218" watchObservedRunningTime="2026-07-13 19:30:20.925621144 +0000 UTC m=+312.487019188" Jul 13 19:30:22.446773 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:22.446741 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:30:23.365512 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:23.365471 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:23.365714 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:23.365544 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:23.370312 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:23.370291 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:23.921091 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:23.921064 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:30:23.961249 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:23.961219 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:27.431062 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.431014 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9"] Jul 13 19:30:27.433972 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.433951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.436314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.436294 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 13 19:30:27.436567 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.436545 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 13 19:30:27.437852 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.437327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xpbnb\"" Jul 13 19:30:27.437852 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.437577 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 19:30:27.447474 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.447453 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9"] Jul 13 19:30:27.474719 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.474695 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jxhd4"] Jul 13 19:30:27.477359 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.477331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.479713 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.479695 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 19:30:27.479848 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.479825 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wgvr4\"" Jul 13 19:30:27.480046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.480028 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 19:30:27.480166 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.479765 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 19:30:27.499578 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.499557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.499681 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.499612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2p7t\" (UniqueName: \"kubernetes.io/projected/f25a3ddf-aec3-4829-88b0-15ea49cef014-kube-api-access-n2p7t\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.499759 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.499737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.499807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.499788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f25a3ddf-aec3-4829-88b0-15ea49cef014-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.600600 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pg7z\" (UniqueName: \"kubernetes.io/projected/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-kube-api-access-2pg7z\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.600785 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.600785 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-root\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.600785 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f25a3ddf-aec3-4829-88b0-15ea49cef014-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.600908 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-tls\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.600908 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.600908 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.600908 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-sys\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.601053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.601053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-metrics-client-ca\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.601053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.600979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2p7t\" (UniqueName: \"kubernetes.io/projected/f25a3ddf-aec3-4829-88b0-15ea49cef014-kube-api-access-n2p7t\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.601053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.601002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-wtmp\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.601053 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.601040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-textfile\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.601949 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.601916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f25a3ddf-aec3-4829-88b0-15ea49cef014-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.603947 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.603894 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.604070 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.604048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f25a3ddf-aec3-4829-88b0-15ea49cef014-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.610393 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.610371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2p7t\" (UniqueName: \"kubernetes.io/projected/f25a3ddf-aec3-4829-88b0-15ea49cef014-kube-api-access-n2p7t\") pod \"openshift-state-metrics-84bd68b896-t84w9\" (UID: \"f25a3ddf-aec3-4829-88b0-15ea49cef014\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.702461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-root\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-tls\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702461 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-sys\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-metrics-client-ca\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702507 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-wtmp\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-textfile\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.702764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pg7z\" (UniqueName: \"kubernetes.io/projected/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-kube-api-access-2pg7z\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703005 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.702844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-root\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.703170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-sys\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.703246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-wtmp\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703945 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.703499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-textfile\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703945 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.703730 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-metrics-client-ca\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.703945 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.703890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-accelerators-collector-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.705752 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.705719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-tls\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.705872 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.705850 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.710570 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.710545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pg7z\" (UniqueName: \"kubernetes.io/projected/1ef930f6-17e0-4b8d-acf0-19be8d6a6579-kube-api-access-2pg7z\") pod \"node-exporter-jxhd4\" (UID: \"1ef930f6-17e0-4b8d-acf0-19be8d6a6579\") " pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.745470 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.745437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" Jul 13 19:30:27.788254 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.788028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jxhd4" Jul 13 19:30:27.800925 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:27.800890 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef930f6_17e0_4b8d_acf0_19be8d6a6579.slice/crio-95aad55ef077fa45eaa36d211e21d3707ac1275fc6d7a782d9fce504d3f3c136 WatchSource:0}: Error finding container 95aad55ef077fa45eaa36d211e21d3707ac1275fc6d7a782d9fce504d3f3c136: Status 404 returned error can't find the container with id 95aad55ef077fa45eaa36d211e21d3707ac1275fc6d7a782d9fce504d3f3c136 Jul 13 19:30:27.907950 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.907919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9"] Jul 13 19:30:27.911133 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:27.911110 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25a3ddf_aec3_4829_88b0_15ea49cef014.slice/crio-6654dba7fd5a4aeb87148d2aac0be23403169b1371c304b4060ddb9e977bb625 WatchSource:0}: Error finding container 6654dba7fd5a4aeb87148d2aac0be23403169b1371c304b4060ddb9e977bb625: Status 404 returned error can't find the container with id 6654dba7fd5a4aeb87148d2aac0be23403169b1371c304b4060ddb9e977bb625 Jul 13 19:30:27.938183 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.938151 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxhd4" event={"ID":"1ef930f6-17e0-4b8d-acf0-19be8d6a6579","Type":"ContainerStarted","Data":"95aad55ef077fa45eaa36d211e21d3707ac1275fc6d7a782d9fce504d3f3c136"} Jul 13 19:30:27.940493 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:27.940328 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" event={"ID":"f25a3ddf-aec3-4829-88b0-15ea49cef014","Type":"ContainerStarted","Data":"6654dba7fd5a4aeb87148d2aac0be23403169b1371c304b4060ddb9e977bb625"} Jul 13 19:30:28.944918 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:28.944887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" event={"ID":"f25a3ddf-aec3-4829-88b0-15ea49cef014","Type":"ContainerStarted","Data":"7929df2308b86edfd613ac0aaadfaf5c9a060c3ac2bd18cce9024e239309fafb"} Jul 13 19:30:28.945335 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:28.944951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" event={"ID":"f25a3ddf-aec3-4829-88b0-15ea49cef014","Type":"ContainerStarted","Data":"8e55744843382615ac295e6ff8eb96fd4a3d6cac1c42bc496f5e006d49b99340"} Jul 13 19:30:29.411824 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.411782 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c7476c988-f9rsq"] Jul 13 19:30:29.415336 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.415315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.417950 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.417912 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 13 19:30:29.417950 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.417929 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 13 19:30:29.418118 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.417984 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-95k8kdch2psnm\"" Jul 13 19:30:29.418118 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.417999 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 13 19:30:29.418118 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.418055 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 13 19:30:29.418239 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.418164 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2km84\"" Jul 13 19:30:29.418341 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.418327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 13 19:30:29.425155 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.425124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c7476c988-f9rsq"] Jul 13 19:30:29.517013 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.516981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517013 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-grpc-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5d6\" (UniqueName: \"kubernetes.io/projected/851b076a-286c-4a39-902d-ec4c2f4b681b-kube-api-access-2c5d6\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851b076a-286c-4a39-902d-ec4c2f4b681b-metrics-client-ca\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.517223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.517198 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617635 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617635 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-grpc-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5d6\" (UniqueName: \"kubernetes.io/projected/851b076a-286c-4a39-902d-ec4c2f4b681b-kube-api-access-2c5d6\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851b076a-286c-4a39-902d-ec4c2f4b681b-metrics-client-ca\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.617893 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.617854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.619067 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.619033 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/851b076a-286c-4a39-902d-ec4c2f4b681b-metrics-client-ca\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.620499 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.620474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.620650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.620625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.620998 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.620984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.621216 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.621192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.621271 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.621252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-grpc-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.621305 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.621257 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/851b076a-286c-4a39-902d-ec4c2f4b681b-secret-thanos-querier-tls\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.630015 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.629988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5d6\" (UniqueName: \"kubernetes.io/projected/851b076a-286c-4a39-902d-ec4c2f4b681b-kube-api-access-2c5d6\") pod \"thanos-querier-c7476c988-f9rsq\" (UID: \"851b076a-286c-4a39-902d-ec4c2f4b681b\") " pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.725605 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.725513 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:29.883640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.883508 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c7476c988-f9rsq"] Jul 13 19:30:29.887097 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:29.887069 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851b076a_286c_4a39_902d_ec4c2f4b681b.slice/crio-7d7eb590cb154ccfd9a52ec21f310aa19b6c1431a123c0f21c78eb704a5e1cf3 WatchSource:0}: Error finding container 7d7eb590cb154ccfd9a52ec21f310aa19b6c1431a123c0f21c78eb704a5e1cf3: Status 404 returned error can't find the container with id 7d7eb590cb154ccfd9a52ec21f310aa19b6c1431a123c0f21c78eb704a5e1cf3 Jul 13 19:30:29.950770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:29.950721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"7d7eb590cb154ccfd9a52ec21f310aa19b6c1431a123c0f21c78eb704a5e1cf3"} Jul 13 19:30:30.957190 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:30.957158 2582 generic.go:356] "Generic (PLEG): container finished" podID="1ef930f6-17e0-4b8d-acf0-19be8d6a6579" containerID="1e897dc74d1512792d051dadac0c4fdeddfd91ec691b29a4586def03e56163bd" exitCode=0 Jul 13 19:30:30.961113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:30.961051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxhd4" event={"ID":"1ef930f6-17e0-4b8d-acf0-19be8d6a6579","Type":"ContainerDied","Data":"1e897dc74d1512792d051dadac0c4fdeddfd91ec691b29a4586def03e56163bd"} Jul 13 19:30:31.743244 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.743208 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54dbd485fd-7c4dc"] Jul 13 19:30:31.747147 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.747125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.749833 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.749810 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 13 19:30:31.750921 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.750898 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ksbg5\"" Jul 13 19:30:31.751040 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.750924 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4cd9eot6b189b\"" Jul 13 19:30:31.751040 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.750926 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 13 19:30:31.751040 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.750950 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 19:30:31.751040 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.750898 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 13 19:30:31.756467 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.756448 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54dbd485fd-7c4dc"] Jul 13 19:30:31.840567 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-tls\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-client-certs\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-metrics-server-audit-profiles\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-client-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-audit-log\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.840958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.840764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5lg\" (UniqueName: \"kubernetes.io/projected/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-kube-api-access-fs5lg\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941667 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5lg\" (UniqueName: \"kubernetes.io/projected/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-kube-api-access-fs5lg\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-tls\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-client-certs\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-metrics-server-audit-profiles\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-client-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.941838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-audit-log\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.942078 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.941873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.942484 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.942406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-audit-log\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.942697 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.942673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.943133 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.943107 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-metrics-server-audit-profiles\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.944792 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.944753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-client-ca-bundle\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.944895 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.944829 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-client-certs\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.944947 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.944905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-secret-metrics-server-tls\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.950527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.950466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5lg\" (UniqueName: \"kubernetes.io/projected/ac9d5c01-b50e-4e05-9a5d-16b6fc276983-kube-api-access-fs5lg\") pod \"metrics-server-54dbd485fd-7c4dc\" (UID: \"ac9d5c01-b50e-4e05-9a5d-16b6fc276983\") " pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:31.969072 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.969023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxhd4" event={"ID":"1ef930f6-17e0-4b8d-acf0-19be8d6a6579","Type":"ContainerStarted","Data":"8400a138965d789ea8da9635fb630b08580ca250448ecedd49eabf31d272e890"} Jul 13 19:30:31.969554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.969141 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jxhd4" event={"ID":"1ef930f6-17e0-4b8d-acf0-19be8d6a6579","Type":"ContainerStarted","Data":"0cbea7f8e5dc5a42af73a9d4c7d8112c553e5ee9364d75342b47e58bba8bc5b4"} Jul 13 19:30:31.971615 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.971589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" event={"ID":"f25a3ddf-aec3-4829-88b0-15ea49cef014","Type":"ContainerStarted","Data":"2d756a9bc58c3fea43ba763838fceb8ea3f373019feed702447632add7c1fa7b"} Jul 13 19:30:31.987562 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:31.987505 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jxhd4" podStartSLOduration=2.5011145790000002 podStartE2EDuration="4.987488601s" totalImagesPullingTime="2.411025935s" totalInitContainerRuntime="75.348087ms" isStatefulPod=true podCreationTimestamp="2026-07-13 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:31.986574254 +0000 UTC m=+323.547972313" watchObservedRunningTime="2026-07-13 19:30:31.987488601 +0000 UTC m=+323.548886646" Jul 13 19:30:32.006499 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.006370 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t84w9" podStartSLOduration=2.081599933 podStartE2EDuration="5.006353856s" totalImagesPullingTime="2.924753923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:32.004824918 +0000 UTC m=+323.566222965" watchObservedRunningTime="2026-07-13 19:30:32.006353856 +0000 UTC m=+323.567751900" Jul 13 19:30:32.058139 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.058106 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:32.225863 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.225713 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:30:32.229316 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.229297 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.235342 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.235323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54dbd485fd-7c4dc"] Jul 13 19:30:32.239019 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:32.238970 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9d5c01_b50e_4e05_9a5d_16b6fc276983.slice/crio-89aab856895191c5237a1fae72e5ed87f5cc2aed6bd44a6370a8826c4cbfdc69 WatchSource:0}: Error finding container 89aab856895191c5237a1fae72e5ed87f5cc2aed6bd44a6370a8826c4cbfdc69: Status 404 returned error can't find the container with id 89aab856895191c5237a1fae72e5ed87f5cc2aed6bd44a6370a8826c4cbfdc69 Jul 13 19:30:32.242846 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.242826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:30:32.345503 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345503 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345734 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345734 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345734 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345734 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpspd\" (UniqueName: \"kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.345880 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.345740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.446838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.446802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.446996 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.446862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.446996 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.446885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.446996 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.446946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.446996 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.446973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.447167 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.447007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.447167 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.447034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpspd\" (UniqueName: \"kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.447678 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.447654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.447798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.447759 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.447859 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.447845 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.448063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.448041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.449469 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.449446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.449557 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.449489 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.454613 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.454590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpspd\" (UniqueName: \"kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd\") pod \"console-fbdb647db-md5mk\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.541216 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.541134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:32.673579 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.673553 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:30:32.676016 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:32.675992 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b3574b_5628_4e4a_944c_182a54800797.slice/crio-b60fbd58d629c21e7bcb01bc1f5123dd5e0dfc3ba17091ac9789aa313e0e25c8 WatchSource:0}: Error finding container b60fbd58d629c21e7bcb01bc1f5123dd5e0dfc3ba17091ac9789aa313e0e25c8: Status 404 returned error can't find the container with id b60fbd58d629c21e7bcb01bc1f5123dd5e0dfc3ba17091ac9789aa313e0e25c8 Jul 13 19:30:32.977939 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.977893 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"d1df6541c41ad4a34011e8aa20633e6ef6f437d82ecef0306751b827a0f5ddbc"} Jul 13 19:30:32.978389 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.977975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"76ac33f4804abe0e9449ad670dec8c5a377b7562392f9d50ef677429c0463d35"} Jul 13 19:30:32.978389 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.978017 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"f695b69558ff96586bfb3eb434fc6dc1f53aa0df22936a10b06c40932f56f48f"} Jul 13 19:30:32.979176 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.979150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" event={"ID":"ac9d5c01-b50e-4e05-9a5d-16b6fc276983","Type":"ContainerStarted","Data":"89aab856895191c5237a1fae72e5ed87f5cc2aed6bd44a6370a8826c4cbfdc69"} Jul 13 19:30:32.980715 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.980691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdb647db-md5mk" event={"ID":"e2b3574b-5628-4e4a-944c-182a54800797","Type":"ContainerStarted","Data":"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b"} Jul 13 19:30:32.980815 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.980729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdb647db-md5mk" event={"ID":"e2b3574b-5628-4e4a-944c-182a54800797","Type":"ContainerStarted","Data":"b60fbd58d629c21e7bcb01bc1f5123dd5e0dfc3ba17091ac9789aa313e0e25c8"} Jul 13 19:30:32.999443 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:32.999382 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fbdb647db-md5mk" podStartSLOduration=0.999370563 podStartE2EDuration="999.370563ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:32.997247417 +0000 UTC m=+324.558645460" watchObservedRunningTime="2026-07-13 19:30:32.999370563 +0000 UTC m=+324.560768607" Jul 13 19:30:33.986645 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:33.986608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" event={"ID":"ac9d5c01-b50e-4e05-9a5d-16b6fc276983","Type":"ContainerStarted","Data":"7b2a83c9551d51162df9dd00054823c31d891e86ccea56aea0c2a57276bbfd30"} Jul 13 19:30:34.004990 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:34.004891 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" podStartSLOduration=1.490757489 podStartE2EDuration="3.004878135s" totalImagesPullingTime="1.514120646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:34.004072587 +0000 UTC m=+325.565470642" watchObservedRunningTime="2026-07-13 19:30:34.004878135 +0000 UTC m=+325.566276157" Jul 13 19:30:35.996580 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:35.996546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"f85b07dedca7b03d29c3faa4703c0fd43a5c2bbf5926cb19ae21de77cb4754fe"} Jul 13 19:30:35.997036 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:35.996606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"9a1515671a3074dd778d57851497843d52231f5dff4611e3afbf7d324e85e00f"} Jul 13 19:30:35.997036 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:35.996633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" event={"ID":"851b076a-286c-4a39-902d-ec4c2f4b681b","Type":"ContainerStarted","Data":"d2c36fd0c78a6362217aab7d684437c12512301a0b6ea9e40b73c9fe63977f6d"} Jul 13 19:30:35.997036 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:35.996683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:36.019050 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.018706 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" podStartSLOduration=2.116351044 podStartE2EDuration="7.01868828s" totalImagesPullingTime="4.902337236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:36.017366497 +0000 UTC m=+327.578764541" watchObservedRunningTime="2026-07-13 19:30:36.01868828 +0000 UTC m=+327.580086327" Jul 13 19:30:36.382879 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.382846 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:30:36.415421 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.415393 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:30:36.418637 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.418611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.428061 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.428039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:30:36.482927 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.482896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483075 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.482930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483075 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.482993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483075 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.483024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gx5\" (UniqueName: \"kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.483074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.483097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.483228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.483135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gx5\" (UniqueName: \"kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584507 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.584551 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.584536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.585223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.585195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.585451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.585250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.585451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.585265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.585451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.585272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.586891 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.586872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.586971 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.586946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.592726 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.592706 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gx5\" (UniqueName: \"kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5\") pod \"console-6847fdfff8-rc6lq\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.727690 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.727597 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:36.856495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:36.856348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:30:36.859226 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:30:36.859201 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b99c112_1e5b_4e39_8447_fbd469d5c2e0.slice/crio-87924ae3357fffe0ee692f1e7c8f5d1f7e37d15930e3fbb55e7e1e8be9b79c9f WatchSource:0}: Error finding container 87924ae3357fffe0ee692f1e7c8f5d1f7e37d15930e3fbb55e7e1e8be9b79c9f: Status 404 returned error can't find the container with id 87924ae3357fffe0ee692f1e7c8f5d1f7e37d15930e3fbb55e7e1e8be9b79c9f Jul 13 19:30:37.002082 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:37.001989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6847fdfff8-rc6lq" event={"ID":"9b99c112-1e5b-4e39-8447-fbd469d5c2e0","Type":"ContainerStarted","Data":"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4"} Jul 13 19:30:37.002082 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:37.002039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6847fdfff8-rc6lq" event={"ID":"9b99c112-1e5b-4e39-8447-fbd469d5c2e0","Type":"ContainerStarted","Data":"87924ae3357fffe0ee692f1e7c8f5d1f7e37d15930e3fbb55e7e1e8be9b79c9f"} Jul 13 19:30:37.008115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:37.008092 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c7476c988-f9rsq" Jul 13 19:30:37.018620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:37.018578 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6847fdfff8-rc6lq" podStartSLOduration=1.0185645 podStartE2EDuration="1.0185645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:30:37.017797469 +0000 UTC m=+328.579195567" watchObservedRunningTime="2026-07-13 19:30:37.0185645 +0000 UTC m=+328.579962543" Jul 13 19:30:42.541440 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:42.541393 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:30:46.728377 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:46.728341 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:46.728757 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:46.728656 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:46.733114 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:46.733096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:47.045920 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.045841 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:30:47.094010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.093981 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:30:47.467512 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.467470 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerName="registry" containerID="cri-o://5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d" gracePeriod=30 Jul 13 19:30:47.702618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.702595 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:30:47.789980 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.789898 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.789980 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.789941 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.789980 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.789962 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78696\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.789980 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.789984 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.790799 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790102 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.790799 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790147 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.790799 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790179 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.790799 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790217 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted\") pod \"68072f52-bbc2-4f9f-9456-dec46034fd0e\" (UID: \"68072f52-bbc2-4f9f-9456-dec46034fd0e\") " Jul 13 19:30:47.790994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790815 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:30:47.790994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.790551 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:30:47.793083 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.793037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:30:47.793193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.793089 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:30:47.793193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.793092 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696" (OuterVolumeSpecName: "kube-api-access-78696") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "kube-api-access-78696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:30:47.793193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.793125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:30:47.793355 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.793257 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:30:47.800700 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.800680 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "68072f52-bbc2-4f9f-9456-dec46034fd0e" (UID: "68072f52-bbc2-4f9f-9456-dec46034fd0e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:30:47.891620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891584 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-certificates\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891613 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-bound-sa-token\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891620 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891623 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68072f52-bbc2-4f9f-9456-dec46034fd0e-trusted-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891832 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891631 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68072f52-bbc2-4f9f-9456-dec46034fd0e-ca-trust-extracted\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891832 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891639 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-registry-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891832 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891648 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-installation-pull-secrets\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891832 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891656 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78696\" (UniqueName: \"kubernetes.io/projected/68072f52-bbc2-4f9f-9456-dec46034fd0e-kube-api-access-78696\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:47.891832 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:47.891664 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68072f52-bbc2-4f9f-9456-dec46034fd0e-image-registry-private-configuration\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:48.048017 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.047924 2582 generic.go:356] "Generic (PLEG): container finished" podID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerID="5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d" exitCode=0 Jul 13 19:30:48.048017 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.047986 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" Jul 13 19:30:48.048225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.048014 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" event={"ID":"68072f52-bbc2-4f9f-9456-dec46034fd0e","Type":"ContainerDied","Data":"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d"} Jul 13 19:30:48.048225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.048067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bc6d5577-t7988" event={"ID":"68072f52-bbc2-4f9f-9456-dec46034fd0e","Type":"ContainerDied","Data":"18bbb513b9bc608ce91068a134c15c310e6accd5726eee2075e8712c6463557a"} Jul 13 19:30:48.048225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.048087 2582 scope.go:117] "RemoveContainer" containerID="5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d" Jul 13 19:30:48.056796 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.056780 2582 scope.go:117] "RemoveContainer" containerID="5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d" Jul 13 19:30:48.057064 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:30:48.057043 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d\": container with ID starting with 5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d not found: ID does not exist" containerID="5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d" Jul 13 19:30:48.057116 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.057074 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d"} err="failed to get container status \"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d\": rpc error: code = NotFound desc = could not find container \"5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d\": container with ID starting with 5a45ad6daef22d798e8dc656476e2fe82e28389fc810e584868947fc13727a4d not found: ID does not exist" Jul 13 19:30:48.069222 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.069199 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:30:48.073499 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.073480 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bc6d5577-t7988"] Jul 13 19:30:48.959385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.959356 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" path="/var/lib/kubelet/pods/68072f52-bbc2-4f9f-9456-dec46034fd0e/volumes" Jul 13 19:30:48.987842 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:48.987799 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c6c44ff65-rbqrl" podUID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" containerName="console" containerID="cri-o://26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e" gracePeriod=15 Jul 13 19:30:49.239115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.239094 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:49.404733 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.404702 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.404733 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.404738 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.404964 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.404780 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.405029 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.404954 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.405029 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.405124 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2gn7\" (UniqueName: \"kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7\") pod \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\" (UID: \"aa83aacd-b1b8-4c16-bd76-a9945f173ddf\") " Jul 13 19:30:49.405303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405274 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:30:49.405418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405301 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config" (OuterVolumeSpecName: "console-config") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:30:49.405418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405366 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:49.405418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405375 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:30:49.405418 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.405385 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:49.407359 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.407335 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7" (OuterVolumeSpecName: "kube-api-access-v2gn7") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "kube-api-access-v2gn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:30:49.407359 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.407354 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:30:49.407564 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.407379 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa83aacd-b1b8-4c16-bd76-a9945f173ddf" (UID: "aa83aacd-b1b8-4c16-bd76-a9945f173ddf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:30:49.505881 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.505811 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:49.505881 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.505836 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-oauth-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:49.505881 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.505845 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2gn7\" (UniqueName: \"kubernetes.io/projected/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-kube-api-access-v2gn7\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:49.505881 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:49.505854 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa83aacd-b1b8-4c16-bd76-a9945f173ddf-console-oauth-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:30:50.057538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.057503 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" containerID="26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e" exitCode=0 Jul 13 19:30:50.057991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.057545 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c44ff65-rbqrl" Jul 13 19:30:50.057991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.057580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c44ff65-rbqrl" event={"ID":"aa83aacd-b1b8-4c16-bd76-a9945f173ddf","Type":"ContainerDied","Data":"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e"} Jul 13 19:30:50.057991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.057626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c44ff65-rbqrl" event={"ID":"aa83aacd-b1b8-4c16-bd76-a9945f173ddf","Type":"ContainerDied","Data":"5f9fc23e0626b974e35ea1b681c9c4ae0d650db0c7efbae6e59e518ddbc46a4a"} Jul 13 19:30:50.057991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.057654 2582 scope.go:117] "RemoveContainer" containerID="26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e" Jul 13 19:30:50.066607 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.066591 2582 scope.go:117] "RemoveContainer" containerID="26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e" Jul 13 19:30:50.066870 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:30:50.066850 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e\": container with ID starting with 26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e not found: ID does not exist" containerID="26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e" Jul 13 19:30:50.066962 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.066877 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e"} err="failed to get container status \"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e\": rpc error: code = NotFound desc = could not find container \"26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e\": container with ID starting with 26cf84323ec14469e10fa85c24ca2ac4c72535f83825018dde27c9eac95e636e not found: ID does not exist" Jul 13 19:30:50.078248 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.078225 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:50.081302 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.081283 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6c44ff65-rbqrl"] Jul 13 19:30:50.960116 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:50.960088 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" path="/var/lib/kubelet/pods/aa83aacd-b1b8-4c16-bd76-a9945f173ddf/volumes" Jul 13 19:30:51.063047 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:51.063020 2582 generic.go:356] "Generic (PLEG): container finished" podID="167e0ad5-cd81-4406-b8da-602edba14d4b" containerID="6fa53572f43e5e53eca65a5b9113e8c35cc10f2ac2ccd406e5a976237dd606f4" exitCode=0 Jul 13 19:30:51.063415 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:51.063095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" event={"ID":"167e0ad5-cd81-4406-b8da-602edba14d4b","Type":"ContainerDied","Data":"6fa53572f43e5e53eca65a5b9113e8c35cc10f2ac2ccd406e5a976237dd606f4"} Jul 13 19:30:51.063499 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:51.063465 2582 scope.go:117] "RemoveContainer" containerID="6fa53572f43e5e53eca65a5b9113e8c35cc10f2ac2ccd406e5a976237dd606f4" Jul 13 19:30:52.058408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:52.058374 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:52.058596 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:52.058460 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:30:52.070237 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:52.070207 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-sqtk9" event={"ID":"167e0ad5-cd81-4406-b8da-602edba14d4b","Type":"ContainerStarted","Data":"5160aa83d20646f07160ec529f049ff7c6c1f9764baa73542424185269f1385d"} Jul 13 19:30:58.095924 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:58.095894 2582 generic.go:356] "Generic (PLEG): container finished" podID="2b46d62f-d1a6-41a7-ab51-0064bb54a1bb" containerID="e2b653cb9239361ba040917f10f9f134ca615aa4b6d7008a59bb0137b5acecaa" exitCode=0 Jul 13 19:30:58.096302 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:58.095965 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" event={"ID":"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb","Type":"ContainerDied","Data":"e2b653cb9239361ba040917f10f9f134ca615aa4b6d7008a59bb0137b5acecaa"} Jul 13 19:30:58.096364 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:58.096308 2582 scope.go:117] "RemoveContainer" containerID="e2b653cb9239361ba040917f10f9f134ca615aa4b6d7008a59bb0137b5acecaa" Jul 13 19:30:59.101927 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:30:59.101894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-g76j9" event={"ID":"2b46d62f-d1a6-41a7-ab51-0064bb54a1bb","Type":"ContainerStarted","Data":"e388a0cdeec07e757f32aa25893491426082b99b740a3d582b311b9703df4f53"} Jul 13 19:31:01.403257 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.403141 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fbdb647db-md5mk" podUID="e2b3574b-5628-4e4a-944c-182a54800797" containerName="console" containerID="cri-o://3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b" gracePeriod=15 Jul 13 19:31:01.646811 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.646782 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:31:01.707456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707364 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707417 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707469 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpspd\" (UniqueName: \"kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707498 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707529 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707552 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707596 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca\") pod \"e2b3574b-5628-4e4a-944c-182a54800797\" (UID: \"e2b3574b-5628-4e4a-944c-182a54800797\") " Jul 13 19:31:01.707895 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707844 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:01.707968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.707928 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:01.708071 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.708043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config" (OuterVolumeSpecName: "console-config") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:01.708141 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.708125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:01.709840 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.709813 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:31:01.709840 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.709825 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd" (OuterVolumeSpecName: "kube-api-access-vpspd") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "kube-api-access-vpspd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:31:01.709976 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.709852 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2b3574b-5628-4e4a-944c-182a54800797" (UID: "e2b3574b-5628-4e4a-944c-182a54800797"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:31:01.808385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808353 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-oauth-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808377 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-console-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808387 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808395 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b3574b-5628-4e4a-944c-182a54800797-trusted-ca-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808404 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-oauth-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808412 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpspd\" (UniqueName: \"kubernetes.io/projected/e2b3574b-5628-4e4a-944c-182a54800797-kube-api-access-vpspd\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:01.808650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:01.808421 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b3574b-5628-4e4a-944c-182a54800797-console-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:02.113913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.113881 2582 generic.go:356] "Generic (PLEG): container finished" podID="e2b3574b-5628-4e4a-944c-182a54800797" containerID="3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b" exitCode=0 Jul 13 19:31:02.114110 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.113932 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdb647db-md5mk" Jul 13 19:31:02.114110 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.113970 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdb647db-md5mk" event={"ID":"e2b3574b-5628-4e4a-944c-182a54800797","Type":"ContainerDied","Data":"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b"} Jul 13 19:31:02.114110 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.114019 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdb647db-md5mk" event={"ID":"e2b3574b-5628-4e4a-944c-182a54800797","Type":"ContainerDied","Data":"b60fbd58d629c21e7bcb01bc1f5123dd5e0dfc3ba17091ac9789aa313e0e25c8"} Jul 13 19:31:02.114110 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.114039 2582 scope.go:117] "RemoveContainer" containerID="3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b" Jul 13 19:31:02.123200 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.123183 2582 scope.go:117] "RemoveContainer" containerID="3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b" Jul 13 19:31:02.123463 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:31:02.123447 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b\": container with ID starting with 3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b not found: ID does not exist" containerID="3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b" Jul 13 19:31:02.123517 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.123473 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b"} err="failed to get container status \"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b\": rpc error: code = NotFound desc = could not find container \"3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b\": container with ID starting with 3858e7c5e1558ad4e564b839c0fe069bf26d7dc842832f693ee8aa18634b210b not found: ID does not exist" Jul 13 19:31:02.133806 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.133780 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:31:02.137570 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.137547 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbdb647db-md5mk"] Jul 13 19:31:02.960200 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:02.960170 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b3574b-5628-4e4a-944c-182a54800797" path="/var/lib/kubelet/pods/e2b3574b-5628-4e4a-944c-182a54800797/volumes" Jul 13 19:31:12.064449 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.064402 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:31:12.070228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.070205 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54dbd485fd-7c4dc" Jul 13 19:31:12.114819 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.114736 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76fc887548-wqgm7" podUID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" containerName="console" containerID="cri-o://039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633" gracePeriod=15 Jul 13 19:31:12.411926 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.411900 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:31:12.502889 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.502851 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhrjg\" (UniqueName: \"kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.502901 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.502979 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503026 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503070 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503101 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503132 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config\") pod \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\" (UID: \"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd\") " Jul 13 19:31:12.503370 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503341 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:12.503536 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503516 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:12.503685 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503650 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config" (OuterVolumeSpecName: "console-config") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:12.503917 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.503729 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:31:12.505413 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.505389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:31:12.505648 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.505619 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:31:12.505753 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.505712 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg" (OuterVolumeSpecName: "kube-api-access-lhrjg") pod "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" (UID: "f3c5c561-83d2-4804-bc8b-20f17ba1d0dd"). InnerVolumeSpecName "kube-api-access-lhrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:31:12.603953 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603915 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.603953 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603947 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-trusted-ca-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.603953 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603960 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-oauth-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.604172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603972 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-oauth-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.604172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603983 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.604172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.603994 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrjg\" (UniqueName: \"kubernetes.io/projected/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-kube-api-access-lhrjg\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:12.604172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:12.604006 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd-console-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:31:13.160570 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.160538 2582 generic.go:356] "Generic (PLEG): container finished" podID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" containerID="039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633" exitCode=0 Jul 13 19:31:13.160961 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.160579 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76fc887548-wqgm7" Jul 13 19:31:13.160961 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.160617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc887548-wqgm7" event={"ID":"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd","Type":"ContainerDied","Data":"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633"} Jul 13 19:31:13.160961 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.160670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76fc887548-wqgm7" event={"ID":"f3c5c561-83d2-4804-bc8b-20f17ba1d0dd","Type":"ContainerDied","Data":"8d74be408cca81974fc1810f51e6e3fdd52a3bd8b4f7336301467ba8943147ab"} Jul 13 19:31:13.160961 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.160697 2582 scope.go:117] "RemoveContainer" containerID="039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633" Jul 13 19:31:13.168570 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.168547 2582 scope.go:117] "RemoveContainer" containerID="039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633" Jul 13 19:31:13.168826 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:31:13.168808 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633\": container with ID starting with 039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633 not found: ID does not exist" containerID="039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633" Jul 13 19:31:13.168889 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.168836 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633"} err="failed to get container status \"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633\": rpc error: code = NotFound desc = could not find container \"039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633\": container with ID starting with 039ad6e53816f91627b0a121a2f9f9911ca35fabcbb8b8445a136b9919b31633 not found: ID does not exist" Jul 13 19:31:13.178840 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.178816 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:31:13.181709 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:13.181693 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76fc887548-wqgm7"] Jul 13 19:31:14.960033 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:14.960004 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" path="/var/lib/kubelet/pods/f3c5c561-83d2-4804-bc8b-20f17ba1d0dd/volumes" Jul 13 19:31:22.894991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:22.894952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:31:22.897339 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:22.897321 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a9254d4-7197-48a0-a784-bb398df08092-metrics-certs\") pod \"network-metrics-daemon-9qx6r\" (UID: \"3a9254d4-7197-48a0-a784-bb398df08092\") " pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:31:23.058121 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:23.058085 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzrsk\"" Jul 13 19:31:23.066129 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:23.066109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qx6r" Jul 13 19:31:23.215077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:23.215055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qx6r"] Jul 13 19:31:23.217843 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:31:23.217820 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9254d4_7197_48a0_a784_bb398df08092.slice/crio-e99db13009bd26cf89b1ff46d10517c60158a91013ac1ca5bd7b5ea018c2e007 WatchSource:0}: Error finding container e99db13009bd26cf89b1ff46d10517c60158a91013ac1ca5bd7b5ea018c2e007: Status 404 returned error can't find the container with id e99db13009bd26cf89b1ff46d10517c60158a91013ac1ca5bd7b5ea018c2e007 Jul 13 19:31:24.203687 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:24.203659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qx6r" event={"ID":"3a9254d4-7197-48a0-a784-bb398df08092","Type":"ContainerStarted","Data":"72e5fdf15689c1c36b0dd8edbbd314f32fe331d2d6f99461b2479becfeedb9cd"} Jul 13 19:31:24.203981 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:24.203724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qx6r" event={"ID":"3a9254d4-7197-48a0-a784-bb398df08092","Type":"ContainerStarted","Data":"e99db13009bd26cf89b1ff46d10517c60158a91013ac1ca5bd7b5ea018c2e007"} Jul 13 19:31:25.209281 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:25.209246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qx6r" event={"ID":"3a9254d4-7197-48a0-a784-bb398df08092","Type":"ContainerStarted","Data":"6c45f888661645b95aeee9c9477985ba731940adebb8117b2cc9289ae4d69c79"} Jul 13 19:31:25.231399 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:25.231343 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9qx6r" podStartSLOduration=374.3879726 podStartE2EDuration="6m15.231323146s" totalImagesPullingTime="843.350546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:31:25.22934401 +0000 UTC m=+376.790742063" watchObservedRunningTime="2026-07-13 19:31:25.231323146 +0000 UTC m=+376.792721192" Jul 13 19:31:46.314497 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314440 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314783 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314795 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314809 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerName="registry" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314818 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerName="registry" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314830 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b3574b-5628-4e4a-944c-182a54800797" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314835 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b3574b-5628-4e4a-944c-182a54800797" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314849 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314854 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314905 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c5c561-83d2-4804-bc8b-20f17ba1d0dd" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314913 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa83aacd-b1b8-4c16-bd76-a9945f173ddf" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314919 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b3574b-5628-4e4a-944c-182a54800797" containerName="console" Jul 13 19:31:46.314922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.314925 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68072f52-bbc2-4f9f-9456-dec46034fd0e" containerName="registry" Jul 13 19:31:46.320882 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.320857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.329358 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.329333 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:31:46.484939 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.484899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.484939 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.484937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.485164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.484965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.485164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.485017 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.485164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.485045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6tbg\" (UniqueName: \"kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.485164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.485106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.485164 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.485149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586307 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586307 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586307 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.586608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.586406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6tbg\" (UniqueName: \"kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.587165 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.587138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.587272 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.587198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.587272 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.587198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.587523 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.587500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.589091 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.589060 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.589181 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.589124 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.594572 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.594553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6tbg\" (UniqueName: \"kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg\") pod \"console-7d6f7ccd87-ghxwd\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.632689 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.632655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:46.774590 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:46.774564 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:31:46.777071 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:31:46.777037 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201c78a_91eb_4799_881f_d350e17cdc61.slice/crio-ac01f88b183800d32305bf8b9c9a2eff9be4764bb1e1341ab1bbb0bdc9e800e4 WatchSource:0}: Error finding container ac01f88b183800d32305bf8b9c9a2eff9be4764bb1e1341ab1bbb0bdc9e800e4: Status 404 returned error can't find the container with id ac01f88b183800d32305bf8b9c9a2eff9be4764bb1e1341ab1bbb0bdc9e800e4 Jul 13 19:31:47.295301 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:47.295270 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6f7ccd87-ghxwd" event={"ID":"0201c78a-91eb-4799-881f-d350e17cdc61","Type":"ContainerStarted","Data":"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f"} Jul 13 19:31:47.295498 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:47.295316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6f7ccd87-ghxwd" event={"ID":"0201c78a-91eb-4799-881f-d350e17cdc61","Type":"ContainerStarted","Data":"ac01f88b183800d32305bf8b9c9a2eff9be4764bb1e1341ab1bbb0bdc9e800e4"} Jul 13 19:31:47.313800 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:47.313751 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6f7ccd87-ghxwd" podStartSLOduration=1.313736063 podStartE2EDuration="1.313736063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:31:47.312057634 +0000 UTC m=+398.873455688" watchObservedRunningTime="2026-07-13 19:31:47.313736063 +0000 UTC m=+398.875134105" Jul 13 19:31:56.633520 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:56.633476 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:56.633903 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:56.633566 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:56.638197 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:56.638173 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:57.337851 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:57.337826 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:31:57.383576 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:31:57.383549 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:32:22.408471 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.408406 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6847fdfff8-rc6lq" podUID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" containerName="console" containerID="cri-o://8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4" gracePeriod=15 Jul 13 19:32:22.644579 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.644557 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:32:22.781063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.780987 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781027 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781064 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781101 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9gx5\" (UniqueName: \"kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781126 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781329 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781148 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config\") pod \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\" (UID: \"9b99c112-1e5b-4e39-8447-fbd469d5c2e0\") " Jul 13 19:32:22.781556 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781525 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:32:22.781556 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781542 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:32:22.781659 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config" (OuterVolumeSpecName: "console-config") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:32:22.781659 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.781567 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:32:22.783452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.783388 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5" (OuterVolumeSpecName: "kube-api-access-v9gx5") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "kube-api-access-v9gx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:32:22.783452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.783402 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:32:22.783452 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.783413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b99c112-1e5b-4e39-8447-fbd469d5c2e0" (UID: "9b99c112-1e5b-4e39-8447-fbd469d5c2e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:32:22.882527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882503 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882523 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882527 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882531 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-trusted-ca-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882539 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882548 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9gx5\" (UniqueName: \"kubernetes.io/projected/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-kube-api-access-v9gx5\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882556 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-oauth-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:22.882718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:22.882563 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b99c112-1e5b-4e39-8447-fbd469d5c2e0-console-oauth-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:32:23.429605 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.429571 2582 generic.go:356] "Generic (PLEG): container finished" podID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" containerID="8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4" exitCode=0 Jul 13 19:32:23.429994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.429615 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6847fdfff8-rc6lq" Jul 13 19:32:23.429994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.429626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6847fdfff8-rc6lq" event={"ID":"9b99c112-1e5b-4e39-8447-fbd469d5c2e0","Type":"ContainerDied","Data":"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4"} Jul 13 19:32:23.429994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.429671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6847fdfff8-rc6lq" event={"ID":"9b99c112-1e5b-4e39-8447-fbd469d5c2e0","Type":"ContainerDied","Data":"87924ae3357fffe0ee692f1e7c8f5d1f7e37d15930e3fbb55e7e1e8be9b79c9f"} Jul 13 19:32:23.429994 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.429691 2582 scope.go:117] "RemoveContainer" containerID="8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4" Jul 13 19:32:23.437738 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.437722 2582 scope.go:117] "RemoveContainer" containerID="8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4" Jul 13 19:32:23.437971 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:32:23.437952 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4\": container with ID starting with 8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4 not found: ID does not exist" containerID="8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4" Jul 13 19:32:23.438031 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.437983 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4"} err="failed to get container status \"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4\": rpc error: code = NotFound desc = could not find container \"8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4\": container with ID starting with 8105b0252462243f9bf0c114e822dab5b67d6921d5cab40f4e56e6e4040f72c4 not found: ID does not exist" Jul 13 19:32:23.446077 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.446058 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:32:23.451751 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:23.451731 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6847fdfff8-rc6lq"] Jul 13 19:32:24.959174 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:24.959140 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" path="/var/lib/kubelet/pods/9b99c112-1e5b-4e39-8447-fbd469d5c2e0/volumes" Jul 13 19:32:54.704160 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.704125 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn"] Jul 13 19:32:54.704631 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.704404 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" containerName="console" Jul 13 19:32:54.704631 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.704414 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" containerName="console" Jul 13 19:32:54.704631 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.704501 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b99c112-1e5b-4e39-8447-fbd469d5c2e0" containerName="console" Jul 13 19:32:54.707666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.707645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.710274 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.710253 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 19:32:54.710388 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.710331 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bhnrj\"" Jul 13 19:32:54.711192 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.711175 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 19:32:54.715952 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.715929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn"] Jul 13 19:32:54.744837 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.744796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvf4\" (UniqueName: \"kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.745030 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.744849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.745030 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.744899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.845476 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.845412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvf4\" (UniqueName: \"kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.845618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.845510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.845618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.845548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.845972 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.845952 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.846010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.845967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:54.854804 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:54.854778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvf4\" (UniqueName: \"kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:55.017287 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:55.017207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:32:55.143305 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:55.143283 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn"] Jul 13 19:32:55.145882 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:32:55.145848 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0e4924_f30e_4009_bc4e_459020e4cda2.slice/crio-ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74 WatchSource:0}: Error finding container ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74: Status 404 returned error can't find the container with id ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74 Jul 13 19:32:55.554291 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:32:55.554258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerStarted","Data":"ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74"} Jul 13 19:33:02.585048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:02.585019 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerStarted","Data":"9d157c23980ba9e0ecbf730b07de53bb5cf029ca4a0ded24d2e83fc40937d6dc"} Jul 13 19:33:03.590745 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:03.590711 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerID="9d157c23980ba9e0ecbf730b07de53bb5cf029ca4a0ded24d2e83fc40937d6dc" exitCode=0 Jul 13 19:33:03.591168 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:03.590757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerDied","Data":"9d157c23980ba9e0ecbf730b07de53bb5cf029ca4a0ded24d2e83fc40937d6dc"} Jul 13 19:33:06.606385 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:06.606355 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerID="4b6b79bc2f117c545606be4b2bd196ffb8e7ce42ee955af279c4507158f6e2a7" exitCode=0 Jul 13 19:33:06.606769 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:06.606419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerDied","Data":"4b6b79bc2f117c545606be4b2bd196ffb8e7ce42ee955af279c4507158f6e2a7"} Jul 13 19:33:15.646486 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:15.646418 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerID="2a3b415a46e7043af987140060054a9b458d30600d72ff6f607787e3afafafd5" exitCode=0 Jul 13 19:33:15.646880 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:15.646532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerDied","Data":"2a3b415a46e7043af987140060054a9b458d30600d72ff6f607787e3afafafd5"} Jul 13 19:33:16.777550 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.777525 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:33:16.825198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.825168 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle\") pod \"ba0e4924-f30e-4009-bc4e-459020e4cda2\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " Jul 13 19:33:16.825373 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.825225 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util\") pod \"ba0e4924-f30e-4009-bc4e-459020e4cda2\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " Jul 13 19:33:16.825373 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.825311 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gvf4\" (UniqueName: \"kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4\") pod \"ba0e4924-f30e-4009-bc4e-459020e4cda2\" (UID: \"ba0e4924-f30e-4009-bc4e-459020e4cda2\") " Jul 13 19:33:16.825900 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.825873 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle" (OuterVolumeSpecName: "bundle") pod "ba0e4924-f30e-4009-bc4e-459020e4cda2" (UID: "ba0e4924-f30e-4009-bc4e-459020e4cda2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:16.827613 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.827583 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4" (OuterVolumeSpecName: "kube-api-access-8gvf4") pod "ba0e4924-f30e-4009-bc4e-459020e4cda2" (UID: "ba0e4924-f30e-4009-bc4e-459020e4cda2"). InnerVolumeSpecName "kube-api-access-8gvf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:33:16.829502 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.829481 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util" (OuterVolumeSpecName: "util") pod "ba0e4924-f30e-4009-bc4e-459020e4cda2" (UID: "ba0e4924-f30e-4009-bc4e-459020e4cda2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:16.926292 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.926189 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvf4\" (UniqueName: \"kubernetes.io/projected/ba0e4924-f30e-4009-bc4e-459020e4cda2-kube-api-access-8gvf4\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:16.926292 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.926235 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:16.926292 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:16.926244 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba0e4924-f30e-4009-bc4e-459020e4cda2-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:17.656145 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:17.656113 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" event={"ID":"ba0e4924-f30e-4009-bc4e-459020e4cda2","Type":"ContainerDied","Data":"ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74"} Jul 13 19:33:17.656308 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:17.656157 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae60026b41b21e70bed9dd5da53d84050963b39c7389cc8be73fdf9a52ddbe74" Jul 13 19:33:17.656308 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:17.656157 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddfpn" Jul 13 19:33:26.821758 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.821723 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b4bvm"] Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822011 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="util" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822020 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="util" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822033 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="pull" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822038 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="pull" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822051 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="extract" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822056 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="extract" Jul 13 19:33:26.822187 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.822105 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0e4924-f30e-4009-bc4e-459020e4cda2" containerName="extract" Jul 13 19:33:26.825838 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.825823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:26.828401 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.828378 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 19:33:26.829264 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.829249 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 19:33:26.829400 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.829268 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-w9rnc\"" Jul 13 19:33:26.839410 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.839382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b4bvm"] Jul 13 19:33:26.907888 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.907856 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84scq\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-kube-api-access-84scq\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:26.908076 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.907895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:26.921501 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.921469 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh"] Jul 13 19:33:26.923856 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.923840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:26.926536 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.926516 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 19:33:26.926536 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.926530 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 19:33:26.926676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.926626 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bhnrj\"" Jul 13 19:33:26.933892 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:26.933870 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh"] Jul 13 19:33:27.008322 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.008291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84scq\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-kube-api-access-84scq\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:27.008322 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.008327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:27.008547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.008382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzs5\" (UniqueName: \"kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.008547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.008413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.008547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.008473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.018437 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.018389 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:27.018545 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.018491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84scq\" (UniqueName: \"kubernetes.io/projected/0ada11ff-0ee2-47d6-a938-8f6886175112-kube-api-access-84scq\") pod \"cert-manager-webhook-6486ffd997-b4bvm\" (UID: \"0ada11ff-0ee2-47d6-a938-8f6886175112\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:27.109856 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.109826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzs5\" (UniqueName: \"kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.110017 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.109864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.110017 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.110000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.110215 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.110199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.110315 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.110299 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.119003 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.118974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzs5\" (UniqueName: \"kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.147906 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.147867 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:27.232691 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.232658 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:27.282355 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.282321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b4bvm"] Jul 13 19:33:27.287203 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:33:27.287168 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ada11ff_0ee2_47d6_a938_8f6886175112.slice/crio-a1a1662c1a32721663e6ce0976705ed528554aa31e1fa78e2a9ef5ed8ba92e1c WatchSource:0}: Error finding container a1a1662c1a32721663e6ce0976705ed528554aa31e1fa78e2a9ef5ed8ba92e1c: Status 404 returned error can't find the container with id a1a1662c1a32721663e6ce0976705ed528554aa31e1fa78e2a9ef5ed8ba92e1c Jul 13 19:33:27.365324 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.365303 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh"] Jul 13 19:33:27.367475 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:33:27.367455 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6cf6e5_3246_4f09_881e_76ad107bfb44.slice/crio-0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868 WatchSource:0}: Error finding container 0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868: Status 404 returned error can't find the container with id 0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868 Jul 13 19:33:27.702523 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.702409 2582 generic.go:356] "Generic (PLEG): container finished" podID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerID="76ce34103587af8f8e72731205596a0c0b1ae4d22439979e7235a663ba271f03" exitCode=0 Jul 13 19:33:27.702523 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.702498 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" event={"ID":"ad6cf6e5-3246-4f09-881e-76ad107bfb44","Type":"ContainerDied","Data":"76ce34103587af8f8e72731205596a0c0b1ae4d22439979e7235a663ba271f03"} Jul 13 19:33:27.702742 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.702555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" event={"ID":"ad6cf6e5-3246-4f09-881e-76ad107bfb44","Type":"ContainerStarted","Data":"0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868"} Jul 13 19:33:27.703659 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:27.703639 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" event={"ID":"0ada11ff-0ee2-47d6-a938-8f6886175112","Type":"ContainerStarted","Data":"a1a1662c1a32721663e6ce0976705ed528554aa31e1fa78e2a9ef5ed8ba92e1c"} Jul 13 19:33:30.720875 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:30.720841 2582 generic.go:356] "Generic (PLEG): container finished" podID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerID="e2f3477d04ff1ac3696f7ac73a9f70fe41a7d0907f184cc12b2d9f6da84cd5b1" exitCode=0 Jul 13 19:33:30.721287 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:30.720929 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" event={"ID":"ad6cf6e5-3246-4f09-881e-76ad107bfb44","Type":"ContainerDied","Data":"e2f3477d04ff1ac3696f7ac73a9f70fe41a7d0907f184cc12b2d9f6da84cd5b1"} Jul 13 19:33:30.722577 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:30.722555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" event={"ID":"0ada11ff-0ee2-47d6-a938-8f6886175112","Type":"ContainerStarted","Data":"6921cfad8dec561881bd84565e0e7c92e0661928a46e3993742cc9aed98d7913"} Jul 13 19:33:30.722708 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:30.722691 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:30.764233 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:30.764186 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" podStartSLOduration=1.554843296 podStartE2EDuration="4.764169242s" totalImagesPullingTime="3.209325946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:30.763140746 +0000 UTC m=+502.324538792" watchObservedRunningTime="2026-07-13 19:33:30.764169242 +0000 UTC m=+502.325567286" Jul 13 19:33:31.728862 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:31.728829 2582 generic.go:356] "Generic (PLEG): container finished" podID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerID="ed3ccafd2292d6ab8b7fb352e5fc04107dd1952ad01afc75bb4c743f4fc2445b" exitCode=0 Jul 13 19:33:31.729269 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:31.728924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" event={"ID":"ad6cf6e5-3246-4f09-881e-76ad107bfb44","Type":"ContainerDied","Data":"ed3ccafd2292d6ab8b7fb352e5fc04107dd1952ad01afc75bb4c743f4fc2445b"} Jul 13 19:33:32.863012 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.862979 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:32.957318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.957289 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util\") pod \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " Jul 13 19:33:32.957498 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.957343 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle\") pod \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " Jul 13 19:33:32.957498 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.957406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrzs5\" (UniqueName: \"kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5\") pod \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\" (UID: \"ad6cf6e5-3246-4f09-881e-76ad107bfb44\") " Jul 13 19:33:32.957822 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.957792 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle" (OuterVolumeSpecName: "bundle") pod "ad6cf6e5-3246-4f09-881e-76ad107bfb44" (UID: "ad6cf6e5-3246-4f09-881e-76ad107bfb44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:32.959735 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.959711 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5" (OuterVolumeSpecName: "kube-api-access-mrzs5") pod "ad6cf6e5-3246-4f09-881e-76ad107bfb44" (UID: "ad6cf6e5-3246-4f09-881e-76ad107bfb44"). InnerVolumeSpecName "kube-api-access-mrzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:33:32.968379 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:32.968350 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util" (OuterVolumeSpecName: "util") pod "ad6cf6e5-3246-4f09-881e-76ad107bfb44" (UID: "ad6cf6e5-3246-4f09-881e-76ad107bfb44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:33.058241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.058144 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:33.058241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.058178 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6cf6e5-3246-4f09-881e-76ad107bfb44-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:33.058241 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.058191 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzs5\" (UniqueName: \"kubernetes.io/projected/ad6cf6e5-3246-4f09-881e-76ad107bfb44-kube-api-access-mrzs5\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:33.739297 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.739273 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" Jul 13 19:33:33.739487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.739271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqptvh" event={"ID":"ad6cf6e5-3246-4f09-881e-76ad107bfb44","Type":"ContainerDied","Data":"0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868"} Jul 13 19:33:33.739487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:33.739394 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad4b116e5103039cd0d727d8e257ff8a53bd09eb2a0fb791b4d6726af1c9868" Jul 13 19:33:34.219919 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.219881 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m"] Jul 13 19:33:34.220640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220620 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="pull" Jul 13 19:33:34.220640 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220643 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="pull" Jul 13 19:33:34.220780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220665 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="util" Jul 13 19:33:34.220780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220674 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="util" Jul 13 19:33:34.220780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220723 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="extract" Jul 13 19:33:34.220780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220732 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="extract" Jul 13 19:33:34.220947 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.220868 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6cf6e5-3246-4f09-881e-76ad107bfb44" containerName="extract" Jul 13 19:33:34.225419 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.225394 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.228182 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.228157 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5bxxn\"" Jul 13 19:33:34.229634 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.229612 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m"] Jul 13 19:33:34.368225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.368187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8h5\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-kube-api-access-sk8h5\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.368406 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.368248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.469026 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.468985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.469208 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.469052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8h5\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-kube-api-access-sk8h5\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.478091 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.478025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.478186 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.478154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8h5\" (UniqueName: \"kubernetes.io/projected/398c5b03-7a33-45db-a453-1fb017302730-kube-api-access-sk8h5\") pod \"cert-manager-cainjector-559cdd8c87-vxx4m\" (UID: \"398c5b03-7a33-45db-a453-1fb017302730\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.536793 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.536756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" Jul 13 19:33:34.666849 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.666826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m"] Jul 13 19:33:34.669309 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:33:34.669286 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod398c5b03_7a33_45db_a453_1fb017302730.slice/crio-630c5412a4ff7c0a99660d4f3fd60862c6edbfca4933f5c29e4451abad7ab5ef WatchSource:0}: Error finding container 630c5412a4ff7c0a99660d4f3fd60862c6edbfca4933f5c29e4451abad7ab5ef: Status 404 returned error can't find the container with id 630c5412a4ff7c0a99660d4f3fd60862c6edbfca4933f5c29e4451abad7ab5ef Jul 13 19:33:34.745748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.745662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" event={"ID":"398c5b03-7a33-45db-a453-1fb017302730","Type":"ContainerStarted","Data":"fa7c7e3036ed5569d6618808070061220d40d40de5ca50509d5de810d84b8a21"} Jul 13 19:33:34.745748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.745701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" event={"ID":"398c5b03-7a33-45db-a453-1fb017302730","Type":"ContainerStarted","Data":"630c5412a4ff7c0a99660d4f3fd60862c6edbfca4933f5c29e4451abad7ab5ef"} Jul 13 19:33:34.765511 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:34.765451 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vxx4m" podStartSLOduration=0.76540588 podStartE2EDuration="765.40588ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:34.764532208 +0000 UTC m=+506.325930255" watchObservedRunningTime="2026-07-13 19:33:34.76540588 +0000 UTC m=+506.326803924" Jul 13 19:33:36.732052 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:36.732023 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b4bvm" Jul 13 19:33:48.644395 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.644356 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk"] Jul 13 19:33:48.649021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.649000 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.651986 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.651961 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bhnrj\"" Jul 13 19:33:48.652098 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.651968 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 19:33:48.652822 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.652809 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 19:33:48.657556 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.657534 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk"] Jul 13 19:33:48.783134 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.783100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.783300 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.783150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.783300 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.783246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbqb\" (UniqueName: \"kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.884681 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.884643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.884844 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.884731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbqb\" (UniqueName: \"kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.884844 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.884770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.885120 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.885102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.885159 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.885101 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.897321 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.897240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbqb\" (UniqueName: \"kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:48.958903 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:48.958873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:49.089494 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:49.089466 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk"] Jul 13 19:33:49.091028 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:33:49.091002 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c06e51_6d75_4f5c_9942_287e4377e5e2.slice/crio-cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4 WatchSource:0}: Error finding container cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4: Status 404 returned error can't find the container with id cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4 Jul 13 19:33:49.806834 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:49.806804 2582 generic.go:356] "Generic (PLEG): container finished" podID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerID="03a35682a8d86c2667a5556bf51e669bcc93b417243f0f55b3e7018e5338a77b" exitCode=0 Jul 13 19:33:49.807203 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:49.806897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" event={"ID":"02c06e51-6d75-4f5c-9942-287e4377e5e2","Type":"ContainerDied","Data":"03a35682a8d86c2667a5556bf51e669bcc93b417243f0f55b3e7018e5338a77b"} Jul 13 19:33:49.807203 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:49.806947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" event={"ID":"02c06e51-6d75-4f5c-9942-287e4377e5e2","Type":"ContainerStarted","Data":"cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4"} Jul 13 19:33:50.813115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:50.813024 2582 generic.go:356] "Generic (PLEG): container finished" podID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerID="190e1bc1f6a27f707b4b8be77ff9adbebb304837d434e03ee0e4734f94ebae54" exitCode=0 Jul 13 19:33:50.813115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:50.813063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" event={"ID":"02c06e51-6d75-4f5c-9942-287e4377e5e2","Type":"ContainerDied","Data":"190e1bc1f6a27f707b4b8be77ff9adbebb304837d434e03ee0e4734f94ebae54"} Jul 13 19:33:51.819153 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:51.819120 2582 generic.go:356] "Generic (PLEG): container finished" podID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerID="19e340da2246e6e87b5dc1abe06849c787428afdcd31173616435626d48bca8e" exitCode=0 Jul 13 19:33:51.819557 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:51.819188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" event={"ID":"02c06e51-6d75-4f5c-9942-287e4377e5e2","Type":"ContainerDied","Data":"19e340da2246e6e87b5dc1abe06849c787428afdcd31173616435626d48bca8e"} Jul 13 19:33:52.945992 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:52.945971 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:53.121948 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.121913 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util\") pod \"02c06e51-6d75-4f5c-9942-287e4377e5e2\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " Jul 13 19:33:53.122105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.121975 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbbqb\" (UniqueName: \"kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb\") pod \"02c06e51-6d75-4f5c-9942-287e4377e5e2\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " Jul 13 19:33:53.122105 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.122023 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle\") pod \"02c06e51-6d75-4f5c-9942-287e4377e5e2\" (UID: \"02c06e51-6d75-4f5c-9942-287e4377e5e2\") " Jul 13 19:33:53.122860 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.122829 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle" (OuterVolumeSpecName: "bundle") pod "02c06e51-6d75-4f5c-9942-287e4377e5e2" (UID: "02c06e51-6d75-4f5c-9942-287e4377e5e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:53.124174 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.124130 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb" (OuterVolumeSpecName: "kube-api-access-dbbqb") pod "02c06e51-6d75-4f5c-9942-287e4377e5e2" (UID: "02c06e51-6d75-4f5c-9942-287e4377e5e2"). InnerVolumeSpecName "kube-api-access-dbbqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:33:53.128161 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.128142 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util" (OuterVolumeSpecName: "util") pod "02c06e51-6d75-4f5c-9942-287e4377e5e2" (UID: "02c06e51-6d75-4f5c-9942-287e4377e5e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:33:53.223310 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.223272 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:53.223310 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.223302 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbqb\" (UniqueName: \"kubernetes.io/projected/02c06e51-6d75-4f5c-9942-287e4377e5e2-kube-api-access-dbbqb\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:53.223310 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.223314 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c06e51-6d75-4f5c-9942-287e4377e5e2-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:33:53.829037 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.829003 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" Jul 13 19:33:53.829037 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.829027 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af22tlk" event={"ID":"02c06e51-6d75-4f5c-9942-287e4377e5e2","Type":"ContainerDied","Data":"cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4"} Jul 13 19:33:53.829299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:33:53.829099 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd166a1f00abb6be3c046cae797cebda99ac0c9ac0b381bf336b2a12301bc2f4" Jul 13 19:34:04.822023 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.821949 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf"] Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822235 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="util" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822247 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="util" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822254 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="extract" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822260 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="extract" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822272 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="pull" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822277 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="pull" Jul 13 19:34:04.822554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.822329 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c06e51-6d75-4f5c-9942-287e4377e5e2" containerName="extract" Jul 13 19:34:04.825008 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.824991 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:04.827506 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.827485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t9khf\"" Jul 13 19:34:04.827506 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.827493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 19:34:04.827691 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.827591 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 19:34:04.827691 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.827621 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 19:34:04.827691 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.827642 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 19:34:04.844275 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.844254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf"] Jul 13 19:34:04.913632 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.913602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-manifests\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:04.913796 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.913639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-tmp\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:04.913796 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.913736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-webhook-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:04.913796 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.913770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:04.913898 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:04.913797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ww99\" (UniqueName: \"kubernetes.io/projected/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-kube-api-access-7ww99\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014540 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ww99\" (UniqueName: \"kubernetes.io/projected/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-kube-api-access-7ww99\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014540 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-manifests\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014781 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-tmp\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014781 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-webhook-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014781 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.014991 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.014968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-manifests\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.017190 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.017166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-tmp\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.017464 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.017415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.017545 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.017419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-webhook-cert\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.029550 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.029526 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ww99\" (UniqueName: \"kubernetes.io/projected/c02d48ba-0a78-4ebc-9d0a-dc4cf449d947-kube-api-access-7ww99\") pod \"opendatahub-operator-controller-manager-699d4944b8-8pmxf\" (UID: \"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947\") " pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.135416 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.135381 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:05.275209 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.275185 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf"] Jul 13 19:34:05.277620 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:05.277593 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02d48ba_0a78_4ebc_9d0a_dc4cf449d947.slice/crio-3df950d813db5bb6d871af8607435f0168c089a11b25cbb39129d4d1dc7623db WatchSource:0}: Error finding container 3df950d813db5bb6d871af8607435f0168c089a11b25cbb39129d4d1dc7623db: Status 404 returned error can't find the container with id 3df950d813db5bb6d871af8607435f0168c089a11b25cbb39129d4d1dc7623db Jul 13 19:34:05.880571 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:05.880536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" event={"ID":"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947","Type":"ContainerStarted","Data":"3df950d813db5bb6d871af8607435f0168c089a11b25cbb39129d4d1dc7623db"} Jul 13 19:34:09.907456 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:09.907405 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" event={"ID":"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947","Type":"ContainerStarted","Data":"6820c4b3ac1d1ae4e035c7781a38553fb9fba8f1935415cbe3ec551e8350bd94"} Jul 13 19:34:10.914014 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:10.913977 2582 generic.go:356] "Generic (PLEG): container finished" podID="c02d48ba-0a78-4ebc-9d0a-dc4cf449d947" containerID="6820c4b3ac1d1ae4e035c7781a38553fb9fba8f1935415cbe3ec551e8350bd94" exitCode=0 Jul 13 19:34:10.914403 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:10.914065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" event={"ID":"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947","Type":"ContainerDied","Data":"6820c4b3ac1d1ae4e035c7781a38553fb9fba8f1935415cbe3ec551e8350bd94"} Jul 13 19:34:11.920498 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:11.920462 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" event={"ID":"c02d48ba-0a78-4ebc-9d0a-dc4cf449d947","Type":"ContainerStarted","Data":"90dc5ff3cf1298346ddc5523cba18c794d0ca1fb75c7fea1a51e589ad8eecd13"} Jul 13 19:34:11.920876 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:11.920607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:11.947261 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:11.947194 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" podStartSLOduration=2.054813358 podStartE2EDuration="7.947171823s" totalImagesPullingTime="4.865573029s" totalInitContainerRuntime="1.026785436s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:11.943789557 +0000 UTC m=+543.505187601" watchObservedRunningTime="2026-07-13 19:34:11.947171823 +0000 UTC m=+543.508569868" Jul 13 19:34:22.928066 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:22.928036 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-699d4944b8-8pmxf" Jul 13 19:34:25.851144 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.851103 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8rkz"] Jul 13 19:34:25.940774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.940731 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8rkz"] Jul 13 19:34:25.940774 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.940773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:25.943543 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.943519 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 19:34:25.943703 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.943571 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4wfjs\"" Jul 13 19:34:25.990446 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.990388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:25.990611 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:25.990450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr579\" (UniqueName: \"kubernetes.io/projected/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-kube-api-access-rr579\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:26.091078 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.091045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:26.091078 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.091082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr579\" (UniqueName: \"kubernetes.io/projected/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-kube-api-access-rr579\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:26.091292 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.091227 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 19:34:26.091327 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.091316 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert podName:e66b7d8e-f0fb-474b-b9c2-1d408f36abe5 nodeName:}" failed. No retries permitted until 2026-07-13 19:34:26.591291497 +0000 UTC m=+558.152689534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert") pod "odh-model-controller-7998c67dc4-f8rkz" (UID: "e66b7d8e-f0fb-474b-b9c2-1d408f36abe5") : secret "odh-model-controller-webhook-cert" not found Jul 13 19:34:26.106108 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.106047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr579\" (UniqueName: \"kubernetes.io/projected/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-kube-api-access-rr579\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:26.489672 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.489593 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sc5j8"] Jul 13 19:34:26.491975 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.491959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:26.494154 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.494127 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 19:34:26.502804 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.502778 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sc5j8"] Jul 13 19:34:26.595591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.595555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:26.595591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.595590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62fq\" (UniqueName: \"kubernetes.io/projected/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-kube-api-access-v62fq\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:26.595787 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.595677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:26.595823 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.595793 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 19:34:26.595863 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.595857 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert podName:e66b7d8e-f0fb-474b-b9c2-1d408f36abe5 nodeName:}" failed. No retries permitted until 2026-07-13 19:34:27.59583812 +0000 UTC m=+559.157236142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert") pod "odh-model-controller-7998c67dc4-f8rkz" (UID: "e66b7d8e-f0fb-474b-b9c2-1d408f36abe5") : secret "odh-model-controller-webhook-cert" not found Jul 13 19:34:26.696981 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.696954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:26.696981 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.696985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62fq\" (UniqueName: \"kubernetes.io/projected/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-kube-api-access-v62fq\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:26.697213 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.697127 2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 19:34:26.697213 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:26.697206 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs podName:84c3e3bc-556c-4cc2-b5da-fa1402cbcca2 nodeName:}" failed. No retries permitted until 2026-07-13 19:34:27.197185945 +0000 UTC m=+558.758583972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs") pod "model-serving-api-54d89bd5fc-sc5j8" (UID: "84c3e3bc-556c-4cc2-b5da-fa1402cbcca2") : secret "model-serving-api-tls" not found Jul 13 19:34:26.705689 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:26.705665 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62fq\" (UniqueName: \"kubernetes.io/projected/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-kube-api-access-v62fq\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:27.201115 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.201072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:27.203583 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.203565 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3e3bc-556c-4cc2-b5da-fa1402cbcca2-tls-certs\") pod \"model-serving-api-54d89bd5fc-sc5j8\" (UID: \"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2\") " pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:27.403355 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.403317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:27.538326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.538295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sc5j8"] Jul 13 19:34:27.539588 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:27.539545 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c3e3bc_556c_4cc2_b5da_fa1402cbcca2.slice/crio-1cdd4e6573ccfc26251aca045a8d26d4745b4988e762e3f3af520887e3ad36ce WatchSource:0}: Error finding container 1cdd4e6573ccfc26251aca045a8d26d4745b4988e762e3f3af520887e3ad36ce: Status 404 returned error can't find the container with id 1cdd4e6573ccfc26251aca045a8d26d4745b4988e762e3f3af520887e3ad36ce Jul 13 19:34:27.604107 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.604079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:27.606658 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.606636 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e66b7d8e-f0fb-474b-b9c2-1d408f36abe5-cert\") pod \"odh-model-controller-7998c67dc4-f8rkz\" (UID: \"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5\") " pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:27.752489 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.752389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:27.888420 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.888392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f8rkz"] Jul 13 19:34:27.890055 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:27.890030 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66b7d8e_f0fb_474b_b9c2_1d408f36abe5.slice/crio-67970f64a6a76097964ec676dffd118fea5331f488aa309d237a50ebfad365f1 WatchSource:0}: Error finding container 67970f64a6a76097964ec676dffd118fea5331f488aa309d237a50ebfad365f1: Status 404 returned error can't find the container with id 67970f64a6a76097964ec676dffd118fea5331f488aa309d237a50ebfad365f1 Jul 13 19:34:27.997177 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.997138 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" event={"ID":"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2","Type":"ContainerStarted","Data":"1cdd4e6573ccfc26251aca045a8d26d4745b4988e762e3f3af520887e3ad36ce"} Jul 13 19:34:27.998256 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:27.998234 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" event={"ID":"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5","Type":"ContainerStarted","Data":"67970f64a6a76097964ec676dffd118fea5331f488aa309d237a50ebfad365f1"} Jul 13 19:34:32.024047 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.024012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" event={"ID":"84c3e3bc-556c-4cc2-b5da-fa1402cbcca2","Type":"ContainerStarted","Data":"216f34c42c98dac0c35c2435a5477326877a4a852b7e1958c55c4eee0d139227"} Jul 13 19:34:32.024547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.024075 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:32.025695 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.025670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" event={"ID":"e66b7d8e-f0fb-474b-b9c2-1d408f36abe5","Type":"ContainerStarted","Data":"c8f6089fc3a4293874513add6f89374d0aad959a3b4d22a2435c95d6e03eb2b0"} Jul 13 19:34:32.025802 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.025736 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:32.041685 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.041633 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" podStartSLOduration=2.584075105 podStartE2EDuration="6.041618858s" totalImagesPullingTime="3.457543753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:32.03989069 +0000 UTC m=+563.601288734" watchObservedRunningTime="2026-07-13 19:34:32.041618858 +0000 UTC m=+563.603016902" Jul 13 19:34:32.057530 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:32.057486 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" podStartSLOduration=3.906463832 podStartE2EDuration="7.057473169s" totalImagesPullingTime="3.151009337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:32.057342746 +0000 UTC m=+563.618740801" watchObservedRunningTime="2026-07-13 19:34:32.057473169 +0000 UTC m=+563.618871217" Jul 13 19:34:35.959463 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.959410 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zhprd"] Jul 13 19:34:35.962733 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.962711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:35.966743 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.966715 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 19:34:35.967043 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.967012 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zbkrx\"" Jul 13 19:34:35.975025 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.975001 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw"] Jul 13 19:34:35.978604 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.978583 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zhprd"] Jul 13 19:34:35.978776 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.978717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:35.980972 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.980952 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 19:34:35.981085 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.981005 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-szct7\"" Jul 13 19:34:35.999948 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:35.999920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw"] Jul 13 19:34:36.083816 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.083776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtnj\" (UniqueName: \"kubernetes.io/projected/de327392-716e-4a5f-b528-f04aa997e9c0-kube-api-access-frtnj\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.084010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.083835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ca9fff-de3e-4b11-a32e-0ab053813325-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.084010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.083895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8884h\" (UniqueName: \"kubernetes.io/projected/92ca9fff-de3e-4b11-a32e-0ab053813325-kube-api-access-8884h\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.084010 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.083929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.184809 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.184779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8884h\" (UniqueName: \"kubernetes.io/projected/92ca9fff-de3e-4b11-a32e-0ab053813325-kube-api-access-8884h\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.184995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.184824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.184995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.184915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtnj\" (UniqueName: \"kubernetes.io/projected/de327392-716e-4a5f-b528-f04aa997e9c0-kube-api-access-frtnj\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.184995 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.184952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ca9fff-de3e-4b11-a32e-0ab053813325-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.185331 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:36.185310 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 19:34:36.185507 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:34:36.185496 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert podName:de327392-716e-4a5f-b528-f04aa997e9c0 nodeName:}" failed. No retries permitted until 2026-07-13 19:34:36.685472696 +0000 UTC m=+568.246870732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert") pod "kserve-controller-manager-d6f4b65b8-zhprd" (UID: "de327392-716e-4a5f-b528-f04aa997e9c0") : secret "kserve-webhook-server-cert" not found Jul 13 19:34:36.190775 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.188126 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92ca9fff-de3e-4b11-a32e-0ab053813325-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.211808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.211740 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8884h\" (UniqueName: \"kubernetes.io/projected/92ca9fff-de3e-4b11-a32e-0ab053813325-kube-api-access-8884h\") pod \"llmisvc-controller-manager-777c6bd7d5-xwpxw\" (UID: \"92ca9fff-de3e-4b11-a32e-0ab053813325\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.242933 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.242906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtnj\" (UniqueName: \"kubernetes.io/projected/de327392-716e-4a5f-b528-f04aa997e9c0-kube-api-access-frtnj\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.289305 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.289269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:36.329308 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.329278 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5"] Jul 13 19:34:36.333556 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.333533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.337574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.337550 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m5s5p\"" Jul 13 19:34:36.337842 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.337807 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 19:34:36.339027 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.338856 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 19:34:36.351920 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.351895 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5"] Jul 13 19:34:36.437172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.437144 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw"] Jul 13 19:34:36.439147 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:36.439109 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92ca9fff_de3e_4b11_a32e_0ab053813325.slice/crio-4aed7b01e9d8e700943c07e508431bca037bd002a64336461ca7949dee9fffc0 WatchSource:0}: Error finding container 4aed7b01e9d8e700943c07e508431bca037bd002a64336461ca7949dee9fffc0: Status 404 returned error can't find the container with id 4aed7b01e9d8e700943c07e508431bca037bd002a64336461ca7949dee9fffc0 Jul 13 19:34:36.486453 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.486345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39706d5-1853-4778-ace2-072e8e80bb0c-tls-certs\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.486453 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.486390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjps\" (UniqueName: \"kubernetes.io/projected/c39706d5-1853-4778-ace2-072e8e80bb0c-kube-api-access-ggjps\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.486674 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.486491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39706d5-1853-4778-ace2-072e8e80bb0c-tmp\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.587333 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.587294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39706d5-1853-4778-ace2-072e8e80bb0c-tls-certs\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.587532 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.587341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjps\" (UniqueName: \"kubernetes.io/projected/c39706d5-1853-4778-ace2-072e8e80bb0c-kube-api-access-ggjps\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.587532 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.587382 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39706d5-1853-4778-ace2-072e8e80bb0c-tmp\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.589746 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.589714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39706d5-1853-4778-ace2-072e8e80bb0c-tmp\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.589860 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.589843 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39706d5-1853-4778-ace2-072e8e80bb0c-tls-certs\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.598717 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.598697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjps\" (UniqueName: \"kubernetes.io/projected/c39706d5-1853-4778-ace2-072e8e80bb0c-kube-api-access-ggjps\") pod \"kube-auth-proxy-6996ff5f69-h7gq5\" (UID: \"c39706d5-1853-4778-ace2-072e8e80bb0c\") " pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.647719 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.647689 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" Jul 13 19:34:36.687917 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.687882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.690777 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.690752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de327392-716e-4a5f-b528-f04aa997e9c0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zhprd\" (UID: \"de327392-716e-4a5f-b528-f04aa997e9c0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:36.779018 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.778992 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5"] Jul 13 19:34:36.780697 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:36.780668 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39706d5_1853_4778_ace2_072e8e80bb0c.slice/crio-2d746820f32a34c025e8602696db1a0125ff64fb55bdf012736beb9b0b90eb3f WatchSource:0}: Error finding container 2d746820f32a34c025e8602696db1a0125ff64fb55bdf012736beb9b0b90eb3f: Status 404 returned error can't find the container with id 2d746820f32a34c025e8602696db1a0125ff64fb55bdf012736beb9b0b90eb3f Jul 13 19:34:36.874770 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:36.874732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:37.060136 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:37.060055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" event={"ID":"92ca9fff-de3e-4b11-a32e-0ab053813325","Type":"ContainerStarted","Data":"4aed7b01e9d8e700943c07e508431bca037bd002a64336461ca7949dee9fffc0"} Jul 13 19:34:37.061256 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:37.061232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" event={"ID":"c39706d5-1853-4778-ace2-072e8e80bb0c","Type":"ContainerStarted","Data":"2d746820f32a34c025e8602696db1a0125ff64fb55bdf012736beb9b0b90eb3f"} Jul 13 19:34:37.071760 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:37.071736 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zhprd"] Jul 13 19:34:37.093106 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:37.093061 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde327392_716e_4a5f_b528_f04aa997e9c0.slice/crio-3905c58e868f43ae15d22c2ddc641dafce4f40f95d1bf018ad9780af966f99e6 WatchSource:0}: Error finding container 3905c58e868f43ae15d22c2ddc641dafce4f40f95d1bf018ad9780af966f99e6: Status 404 returned error can't find the container with id 3905c58e868f43ae15d22c2ddc641dafce4f40f95d1bf018ad9780af966f99e6 Jul 13 19:34:38.070732 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.070680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" event={"ID":"de327392-716e-4a5f-b528-f04aa997e9c0","Type":"ContainerStarted","Data":"3905c58e868f43ae15d22c2ddc641dafce4f40f95d1bf018ad9780af966f99e6"} Jul 13 19:34:38.272242 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.271997 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj"] Jul 13 19:34:38.276761 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.276732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.280633 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.280601 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 19:34:38.280880 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.280860 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lhrcc\"" Jul 13 19:34:38.280962 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.280890 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 19:34:38.281118 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.281102 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 19:34:38.281193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.281150 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 19:34:38.288257 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.288231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj"] Jul 13 19:34:38.403417 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403511 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nlm\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-kube-api-access-84nlm\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403618 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51549055-9645-4fac-b63d-24323f766e19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403856 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.403856 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.403683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51549055-9645-4fac-b63d-24323f766e19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.504850 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.504807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nlm\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-kube-api-access-84nlm\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505001 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.504858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51549055-9645-4fac-b63d-24323f766e19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505001 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.504909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505001 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.504958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51549055-9645-4fac-b63d-24323f766e19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505142 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.505023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505142 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.505056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505142 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.505089 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.505628 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.505602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51549055-9645-4fac-b63d-24323f766e19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.508494 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.508298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.508494 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.508447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.509009 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.508963 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51549055-9645-4fac-b63d-24323f766e19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.509331 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.509306 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51549055-9645-4fac-b63d-24323f766e19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.514088 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.514061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nlm\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-kube-api-access-84nlm\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.514642 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.514624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51549055-9645-4fac-b63d-24323f766e19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6lshj\" (UID: \"51549055-9645-4fac-b63d-24323f766e19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.592262 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.592226 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:38.787873 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:38.787852 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj"] Jul 13 19:34:38.789647 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:34:38.789616 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51549055_9645_4fac_b63d_24323f766e19.slice/crio-e6c1aa8352f55d27c13997e20a55fb7f0e74bcb73da5deccdb032901be30b943 WatchSource:0}: Error finding container e6c1aa8352f55d27c13997e20a55fb7f0e74bcb73da5deccdb032901be30b943: Status 404 returned error can't find the container with id e6c1aa8352f55d27c13997e20a55fb7f0e74bcb73da5deccdb032901be30b943 Jul 13 19:34:39.079894 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:39.079803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" event={"ID":"51549055-9645-4fac-b63d-24323f766e19","Type":"ContainerStarted","Data":"e6c1aa8352f55d27c13997e20a55fb7f0e74bcb73da5deccdb032901be30b943"} Jul 13 19:34:42.099930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.099878 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" event={"ID":"de327392-716e-4a5f-b528-f04aa997e9c0","Type":"ContainerStarted","Data":"881bfadbeb4dfe1f1d28bcd2ddc7dcfb72f73aec60f84e434889da17bbac6416"} Jul 13 19:34:42.100398 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.099970 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:34:42.101823 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.101795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" event={"ID":"92ca9fff-de3e-4b11-a32e-0ab053813325","Type":"ContainerStarted","Data":"229cabf8225db5321fb942a4705607607baf19ea1dcc6d7cc9205080142b092b"} Jul 13 19:34:42.101960 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.101848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:34:42.103446 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.103406 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" event={"ID":"c39706d5-1853-4778-ace2-072e8e80bb0c","Type":"ContainerStarted","Data":"c46960b14d60563cc543c7620a0da8141d3ba561ff39cd5cfd147e69561c0d46"} Jul 13 19:34:42.119961 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.119910 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" podStartSLOduration=3.192186278 podStartE2EDuration="7.119890531s" totalImagesPullingTime="3.927704253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:42.116498842 +0000 UTC m=+573.677896885" watchObservedRunningTime="2026-07-13 19:34:42.119890531 +0000 UTC m=+573.681288576" Jul 13 19:34:42.133080 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.133025 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" podStartSLOduration=2.547304598 podStartE2EDuration="7.133008398s" totalImagesPullingTime="4.5857038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:42.132220568 +0000 UTC m=+573.693618619" watchObservedRunningTime="2026-07-13 19:34:42.133008398 +0000 UTC m=+573.694406446" Jul 13 19:34:42.149135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.149081 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6996ff5f69-h7gq5" podStartSLOduration=1.912225862 podStartE2EDuration="6.149054959s" totalImagesPullingTime="4.236829097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:42.148599941 +0000 UTC m=+573.709998009" watchObservedRunningTime="2026-07-13 19:34:42.149054959 +0000 UTC m=+573.710453003" Jul 13 19:34:42.409608 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.409564 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 13 19:34:42.409698 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:42.409639 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 13 19:34:43.034123 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.034085 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f8rkz" Jul 13 19:34:43.039413 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.039366 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sc5j8" Jul 13 19:34:43.113336 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.113195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" event={"ID":"51549055-9645-4fac-b63d-24323f766e19","Type":"ContainerStarted","Data":"3933a5374cdef46e35c2505b392291eecb3c2eaac8d327055d08203ed6208c93"} Jul 13 19:34:43.113942 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.113921 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:34:43.115394 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.115369 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-6lshj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 19:34:43.115547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.115438 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" podUID="51549055-9645-4fac-b63d-24323f766e19" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 19:34:43.141228 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:43.141177 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" podStartSLOduration=1.5237122360000002 podStartE2EDuration="5.141159988s" totalImagesPullingTime="3.617447752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:34:43.1393734 +0000 UTC m=+574.700771449" watchObservedRunningTime="2026-07-13 19:34:43.141159988 +0000 UTC m=+574.702558031" Jul 13 19:34:44.118353 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:34:44.118327 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6lshj" Jul 13 19:35:08.875048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:08.875023 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:35:08.875641 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:08.875089 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:35:13.115980 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:13.115949 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xwpxw" Jul 13 19:35:13.118728 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:13.118707 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zhprd" Jul 13 19:35:42.237887 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.237794 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf"] Jul 13 19:35:42.240709 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.240687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.243226 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.243203 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vgxbs\"" Jul 13 19:35:42.243336 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.243242 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 19:35:42.243336 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.243315 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 19:35:42.249121 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.249098 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf"] Jul 13 19:35:42.369526 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.369487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65zb\" (UniqueName: \"kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.369730 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.369547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.369730 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.369683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.470144 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.470109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.470301 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.470168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.470301 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.470199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65zb\" (UniqueName: \"kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.470498 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.470477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.470538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.470515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.479035 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.479012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65zb\" (UniqueName: \"kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.551374 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.551313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:42.683515 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.683493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf"] Jul 13 19:35:42.685362 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:42.685334 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522bab15_3c3b_4c40_bcd7_2ace49a7c3d4.slice/crio-0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7 WatchSource:0}: Error finding container 0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7: Status 404 returned error can't find the container with id 0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7 Jul 13 19:35:42.840676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.840607 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k"] Jul 13 19:35:42.843663 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.843638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:42.852641 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.852216 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k"] Jul 13 19:35:42.974184 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.974159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:42.974353 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.974214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxslx\" (UniqueName: \"kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:42.974353 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:42.974250 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.075674 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.075627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxslx\" (UniqueName: \"kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.075862 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.075683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.075862 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.075756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.076067 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.076046 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.076135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.076093 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.084566 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.084543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxslx\" (UniqueName: \"kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.155022 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.154994 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:43.244462 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.244414 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v"] Jul 13 19:35:43.251211 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.251183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.258444 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.258374 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v"] Jul 13 19:35:43.287106 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.287086 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k"] Jul 13 19:35:43.289032 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:43.289008 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d19676_1cae_45cc_8b7a_70abca3dd1d6.slice/crio-ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3 WatchSource:0}: Error finding container ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3: Status 404 returned error can't find the container with id ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3 Jul 13 19:35:43.379051 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.379027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.379171 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.379061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l2x\" (UniqueName: \"kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.379234 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.379203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.389934 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.389908 2582 generic.go:356] "Generic (PLEG): container finished" podID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerID="8c4d1111f385480d745733399273148268ec0eef3630723327ca8b57007554c3" exitCode=0 Jul 13 19:35:43.390050 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.389989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" event={"ID":"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4","Type":"ContainerDied","Data":"8c4d1111f385480d745733399273148268ec0eef3630723327ca8b57007554c3"} Jul 13 19:35:43.390050 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.390047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" event={"ID":"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4","Type":"ContainerStarted","Data":"0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7"} Jul 13 19:35:43.390447 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.390411 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 19:35:43.391798 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.391777 2582 generic.go:356] "Generic (PLEG): container finished" podID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerID="1e7106bf843ab3f851a145df205de0e19e9d1f2094309c1ff23f1eff8909081b" exitCode=0 Jul 13 19:35:43.391870 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.391844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" event={"ID":"23d19676-1cae-45cc-8b7a-70abca3dd1d6","Type":"ContainerDied","Data":"1e7106bf843ab3f851a145df205de0e19e9d1f2094309c1ff23f1eff8909081b"} Jul 13 19:35:43.391917 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.391885 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" event={"ID":"23d19676-1cae-45cc-8b7a-70abca3dd1d6","Type":"ContainerStarted","Data":"ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3"} Jul 13 19:35:43.480987 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.480889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.480987 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.480966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.481200 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.480997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l2x\" (UniqueName: \"kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.481342 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.481323 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.481408 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.481384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.489745 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.489723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l2x\" (UniqueName: \"kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.562599 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.562563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:43.648048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.648016 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8"] Jul 13 19:35:43.660300 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.660270 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8"] Jul 13 19:35:43.660501 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.660390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.682661 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.682623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjbh\" (UniqueName: \"kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.682830 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.682677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.682830 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.682730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.706455 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.706414 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v"] Jul 13 19:35:43.707853 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:43.707826 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d77cd_4578_4afa_a15a_abf9448ac330.slice/crio-61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c WatchSource:0}: Error finding container 61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c: Status 404 returned error can't find the container with id 61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c Jul 13 19:35:43.783917 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.783884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjbh\" (UniqueName: \"kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.784080 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.783931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.784080 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.783954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.784292 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.784275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.784358 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.784299 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.792780 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.792753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjbh\" (UniqueName: \"kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:43.974366 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:43.974334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:44.115626 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.115599 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8"] Jul 13 19:35:44.133007 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:44.132970 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7433e734_b7e5_4112_9f00_ab13a6d97f94.slice/crio-170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b WatchSource:0}: Error finding container 170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b: Status 404 returned error can't find the container with id 170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b Jul 13 19:35:44.398401 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.398368 2582 generic.go:356] "Generic (PLEG): container finished" podID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerID="9225d08fc347f8e80ae7d38bd9d0546405257e0edd4ab9a1b57e72af7c98e910" exitCode=0 Jul 13 19:35:44.398778 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.398453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" event={"ID":"23d19676-1cae-45cc-8b7a-70abca3dd1d6","Type":"ContainerDied","Data":"9225d08fc347f8e80ae7d38bd9d0546405257e0edd4ab9a1b57e72af7c98e910"} Jul 13 19:35:44.400161 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.400136 2582 generic.go:356] "Generic (PLEG): container finished" podID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerID="4bbc23625afc8612614e05d3478421fbf4aef91206d94373b74a214830bfb94d" exitCode=0 Jul 13 19:35:44.400240 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.400196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" event={"ID":"7433e734-b7e5-4112-9f00-ab13a6d97f94","Type":"ContainerDied","Data":"4bbc23625afc8612614e05d3478421fbf4aef91206d94373b74a214830bfb94d"} Jul 13 19:35:44.400298 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.400241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" event={"ID":"7433e734-b7e5-4112-9f00-ab13a6d97f94","Type":"ContainerStarted","Data":"170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b"} Jul 13 19:35:44.401854 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.401833 2582 generic.go:356] "Generic (PLEG): container finished" podID="675d77cd-4578-4afa-a15a-abf9448ac330" containerID="34cee6bb56dc60249d5d7569805706c258b3db85b079d9acc36dfd9f2f6e124f" exitCode=0 Jul 13 19:35:44.401958 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.401914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" event={"ID":"675d77cd-4578-4afa-a15a-abf9448ac330","Type":"ContainerDied","Data":"34cee6bb56dc60249d5d7569805706c258b3db85b079d9acc36dfd9f2f6e124f"} Jul 13 19:35:44.402044 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.401968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" event={"ID":"675d77cd-4578-4afa-a15a-abf9448ac330","Type":"ContainerStarted","Data":"61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c"} Jul 13 19:35:44.404135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.404105 2582 generic.go:356] "Generic (PLEG): container finished" podID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerID="11b66fec1ffe888249c6a2d4e664c91d520f9e698d07cb34b73a8ecce812e7f4" exitCode=0 Jul 13 19:35:44.404135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:44.404130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" event={"ID":"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4","Type":"ContainerDied","Data":"11b66fec1ffe888249c6a2d4e664c91d520f9e698d07cb34b73a8ecce812e7f4"} Jul 13 19:35:45.413314 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.413278 2582 generic.go:356] "Generic (PLEG): container finished" podID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerID="09997ec6afcff4ef471ca5b92c897f52e4abf0545a0d86c724bf3b33937552c6" exitCode=0 Jul 13 19:35:45.413718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.413345 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" event={"ID":"23d19676-1cae-45cc-8b7a-70abca3dd1d6","Type":"ContainerDied","Data":"09997ec6afcff4ef471ca5b92c897f52e4abf0545a0d86c724bf3b33937552c6"} Jul 13 19:35:45.415286 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.415261 2582 generic.go:356] "Generic (PLEG): container finished" podID="675d77cd-4578-4afa-a15a-abf9448ac330" containerID="0c2b290e404d861c179b4db76d216771a2ba9d7d3499ca6c7f695a21f644be9d" exitCode=0 Jul 13 19:35:45.415410 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.415338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" event={"ID":"675d77cd-4578-4afa-a15a-abf9448ac330","Type":"ContainerDied","Data":"0c2b290e404d861c179b4db76d216771a2ba9d7d3499ca6c7f695a21f644be9d"} Jul 13 19:35:45.417669 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.417647 2582 generic.go:356] "Generic (PLEG): container finished" podID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerID="217cc8f69f32fd4c05e16f01aa34a9b752528b83c4401bd600745b6b75682419" exitCode=0 Jul 13 19:35:45.417748 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:45.417721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" event={"ID":"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4","Type":"ContainerDied","Data":"217cc8f69f32fd4c05e16f01aa34a9b752528b83c4401bd600745b6b75682419"} Jul 13 19:35:46.425205 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.425170 2582 generic.go:356] "Generic (PLEG): container finished" podID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerID="9849dff9b287c383407f648274de141fc95a59b13b2eb8a4095fd9ebd63b1a23" exitCode=0 Jul 13 19:35:46.425637 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.425225 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" event={"ID":"7433e734-b7e5-4112-9f00-ab13a6d97f94","Type":"ContainerDied","Data":"9849dff9b287c383407f648274de141fc95a59b13b2eb8a4095fd9ebd63b1a23"} Jul 13 19:35:46.427609 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.427584 2582 generic.go:356] "Generic (PLEG): container finished" podID="675d77cd-4578-4afa-a15a-abf9448ac330" containerID="1daed689f333f3ed6665440586b266f901b3ac101dd53507a1bbf543125465d9" exitCode=0 Jul 13 19:35:46.427697 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.427631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" event={"ID":"675d77cd-4578-4afa-a15a-abf9448ac330","Type":"ContainerDied","Data":"1daed689f333f3ed6665440586b266f901b3ac101dd53507a1bbf543125465d9"} Jul 13 19:35:46.578538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.578512 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:46.605801 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.605779 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle\") pod \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " Jul 13 19:35:46.605916 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.605896 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util\") pod \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " Jul 13 19:35:46.606041 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.605939 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxslx\" (UniqueName: \"kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx\") pod \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\" (UID: \"23d19676-1cae-45cc-8b7a-70abca3dd1d6\") " Jul 13 19:35:46.606450 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.606276 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle" (OuterVolumeSpecName: "bundle") pod "23d19676-1cae-45cc-8b7a-70abca3dd1d6" (UID: "23d19676-1cae-45cc-8b7a-70abca3dd1d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:46.608193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.608166 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx" (OuterVolumeSpecName: "kube-api-access-kxslx") pod "23d19676-1cae-45cc-8b7a-70abca3dd1d6" (UID: "23d19676-1cae-45cc-8b7a-70abca3dd1d6"). InnerVolumeSpecName "kube-api-access-kxslx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:35:46.611533 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.611502 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util" (OuterVolumeSpecName: "util") pod "23d19676-1cae-45cc-8b7a-70abca3dd1d6" (UID: "23d19676-1cae-45cc-8b7a-70abca3dd1d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:46.673845 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.673821 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:46.707316 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707246 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle\") pod \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " Jul 13 19:35:46.707316 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707298 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util\") pod \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " Jul 13 19:35:46.707491 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707368 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m65zb\" (UniqueName: \"kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb\") pod \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\" (UID: \"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4\") " Jul 13 19:35:46.707656 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707638 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:46.707704 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707664 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxslx\" (UniqueName: \"kubernetes.io/projected/23d19676-1cae-45cc-8b7a-70abca3dd1d6-kube-api-access-kxslx\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:46.707704 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707678 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23d19676-1cae-45cc-8b7a-70abca3dd1d6-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:46.707771 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.707721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle" (OuterVolumeSpecName: "bundle") pod "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" (UID: "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:46.709577 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.709550 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb" (OuterVolumeSpecName: "kube-api-access-m65zb") pod "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" (UID: "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4"). InnerVolumeSpecName "kube-api-access-m65zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:35:46.716197 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.716163 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util" (OuterVolumeSpecName: "util") pod "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" (UID: "522bab15-3c3b-4c40-bcd7-2ace49a7c3d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:46.808612 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.808578 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:46.808612 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.808605 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:46.808789 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:46.808618 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65zb\" (UniqueName: \"kubernetes.io/projected/522bab15-3c3b-4c40-bcd7-2ace49a7c3d4-kube-api-access-m65zb\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:47.436396 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.436367 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" Jul 13 19:35:47.436396 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.436383 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k" event={"ID":"23d19676-1cae-45cc-8b7a-70abca3dd1d6","Type":"ContainerDied","Data":"ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3"} Jul 13 19:35:47.436888 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.436451 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad19a2e1a9842f9960972623540556d743a1bc11496c8902262c6466f7ae8db3" Jul 13 19:35:47.438621 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.438594 2582 generic.go:356] "Generic (PLEG): container finished" podID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerID="db134c985d64b44e24f42ba5b1a3fcd272997f0b6e8b8838168350bafbd42edf" exitCode=0 Jul 13 19:35:47.438755 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.438710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" event={"ID":"7433e734-b7e5-4112-9f00-ab13a6d97f94","Type":"ContainerDied","Data":"db134c985d64b44e24f42ba5b1a3fcd272997f0b6e8b8838168350bafbd42edf"} Jul 13 19:35:47.440475 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.440452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" event={"ID":"522bab15-3c3b-4c40-bcd7-2ace49a7c3d4","Type":"ContainerDied","Data":"0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7"} Jul 13 19:35:47.440475 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.440472 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf" Jul 13 19:35:47.440593 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.440492 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0295b1662b768491af39aaba131c648252305fa641034d675292137212d080b7" Jul 13 19:35:47.568143 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.568122 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:47.614585 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.614552 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9l2x\" (UniqueName: \"kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x\") pod \"675d77cd-4578-4afa-a15a-abf9448ac330\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " Jul 13 19:35:47.614735 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.614606 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle\") pod \"675d77cd-4578-4afa-a15a-abf9448ac330\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " Jul 13 19:35:47.614735 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.614649 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util\") pod \"675d77cd-4578-4afa-a15a-abf9448ac330\" (UID: \"675d77cd-4578-4afa-a15a-abf9448ac330\") " Jul 13 19:35:47.615225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.615205 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle" (OuterVolumeSpecName: "bundle") pod "675d77cd-4578-4afa-a15a-abf9448ac330" (UID: "675d77cd-4578-4afa-a15a-abf9448ac330"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:47.616755 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.616736 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x" (OuterVolumeSpecName: "kube-api-access-g9l2x") pod "675d77cd-4578-4afa-a15a-abf9448ac330" (UID: "675d77cd-4578-4afa-a15a-abf9448ac330"). InnerVolumeSpecName "kube-api-access-g9l2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:35:47.620026 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.620002 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util" (OuterVolumeSpecName: "util") pod "675d77cd-4578-4afa-a15a-abf9448ac330" (UID: "675d77cd-4578-4afa-a15a-abf9448ac330"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:47.715251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.715168 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9l2x\" (UniqueName: \"kubernetes.io/projected/675d77cd-4578-4afa-a15a-abf9448ac330-kube-api-access-g9l2x\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:47.715251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.715196 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:47.715251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:47.715205 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675d77cd-4578-4afa-a15a-abf9448ac330-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:48.448207 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.448174 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" Jul 13 19:35:48.448207 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.448191 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v" event={"ID":"675d77cd-4578-4afa-a15a-abf9448ac330","Type":"ContainerDied","Data":"61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c"} Jul 13 19:35:48.448676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.448236 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ed8f570cfb2ee4e4595618b3f19d55ea562b7a1ec28673075e3a05d6e7fa7c" Jul 13 19:35:48.581499 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.581475 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:48.621439 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.621407 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util\") pod \"7433e734-b7e5-4112-9f00-ab13a6d97f94\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " Jul 13 19:35:48.621592 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.621485 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjbh\" (UniqueName: \"kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh\") pod \"7433e734-b7e5-4112-9f00-ab13a6d97f94\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " Jul 13 19:35:48.621592 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.621503 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle\") pod \"7433e734-b7e5-4112-9f00-ab13a6d97f94\" (UID: \"7433e734-b7e5-4112-9f00-ab13a6d97f94\") " Jul 13 19:35:48.622033 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.622009 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle" (OuterVolumeSpecName: "bundle") pod "7433e734-b7e5-4112-9f00-ab13a6d97f94" (UID: "7433e734-b7e5-4112-9f00-ab13a6d97f94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:48.623616 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.623593 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh" (OuterVolumeSpecName: "kube-api-access-wtjbh") pod "7433e734-b7e5-4112-9f00-ab13a6d97f94" (UID: "7433e734-b7e5-4112-9f00-ab13a6d97f94"). InnerVolumeSpecName "kube-api-access-wtjbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:35:48.626516 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.626490 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util" (OuterVolumeSpecName: "util") pod "7433e734-b7e5-4112-9f00-ab13a6d97f94" (UID: "7433e734-b7e5-4112-9f00-ab13a6d97f94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 19:35:48.722068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.722000 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-util\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:48.722068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.722024 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjbh\" (UniqueName: \"kubernetes.io/projected/7433e734-b7e5-4112-9f00-ab13a6d97f94-kube-api-access-wtjbh\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:48.722068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:48.722033 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7433e734-b7e5-4112-9f00-ab13a6d97f94-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:35:49.454605 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:49.454577 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" Jul 13 19:35:49.454971 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:49.454576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8" event={"ID":"7433e734-b7e5-4112-9f00-ab13a6d97f94","Type":"ContainerDied","Data":"170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b"} Jul 13 19:35:49.454971 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:49.454709 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170ce60981d7285cf22a7ca1df92c2f9cadfdb7cca78be651ce86c854a25cc7b" Jul 13 19:35:55.270937 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.270905 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b798ddc6c-jlkxl"] Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271247 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="extract" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271257 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="extract" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271270 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="util" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271275 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="util" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271286 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="pull" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271291 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="pull" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271297 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="extract" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271303 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="extract" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271309 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="util" Jul 13 19:35:55.271311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271314 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="util" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271321 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271327 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271335 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271340 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271345 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271350 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271360 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="util" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271364 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="util" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271370 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271374 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271383 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="util" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271388 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="util" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271393 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271398 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="pull" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271462 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7433e734-b7e5-4112-9f00-ab13a6d97f94" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271471 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="675d77cd-4578-4afa-a15a-abf9448ac330" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271476 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="522bab15-3c3b-4c40-bcd7-2ace49a7c3d4" containerName="extract" Jul 13 19:35:55.271625 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.271481 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d19676-1cae-45cc-8b7a-70abca3dd1d6" containerName="extract" Jul 13 19:35:55.274036 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.274017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.322303 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.322245 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b798ddc6c-jlkxl"] Jul 13 19:35:55.390190 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-oauth-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390190 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390204 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-service-ca\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xkr\" (UniqueName: \"kubernetes.io/projected/ed605944-4582-4d0f-96cc-a073aa8d99ef-kube-api-access-99xkr\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390357 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-trusted-ca-bundle\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.390487 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.390414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-oauth-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490805 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-oauth-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-service-ca\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xkr\" (UniqueName: \"kubernetes.io/projected/ed605944-4582-4d0f-96cc-a073aa8d99ef-kube-api-access-99xkr\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-trusted-ca-bundle\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.490968 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.490945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-oauth-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.491795 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.491771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.491795 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.491781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-oauth-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.491931 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.491821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-service-ca\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.491931 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.491855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed605944-4582-4d0f-96cc-a073aa8d99ef-trusted-ca-bundle\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.493437 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.493403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-oauth-config\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.493574 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.493556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed605944-4582-4d0f-96cc-a073aa8d99ef-console-serving-cert\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.499865 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.499846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xkr\" (UniqueName: \"kubernetes.io/projected/ed605944-4582-4d0f-96cc-a073aa8d99ef-kube-api-access-99xkr\") pod \"console-5b798ddc6c-jlkxl\" (UID: \"ed605944-4582-4d0f-96cc-a073aa8d99ef\") " pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.590131 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.590042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:35:55.735277 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:55.735253 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b798ddc6c-jlkxl"] Jul 13 19:35:55.736976 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:55.736953 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded605944_4582_4d0f_96cc_a073aa8d99ef.slice/crio-b6e8607d18bab92c204587b0fcdc1399e62ac2e9a07d0f63e5939f3aff4dad31 WatchSource:0}: Error finding container b6e8607d18bab92c204587b0fcdc1399e62ac2e9a07d0f63e5939f3aff4dad31: Status 404 returned error can't find the container with id b6e8607d18bab92c204587b0fcdc1399e62ac2e9a07d0f63e5939f3aff4dad31 Jul 13 19:35:56.490106 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:56.490071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b798ddc6c-jlkxl" event={"ID":"ed605944-4582-4d0f-96cc-a073aa8d99ef","Type":"ContainerStarted","Data":"cd936cb55ece99913bce34971322e33fd394b3f235a88f24e2449a97da173710"} Jul 13 19:35:56.490531 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:56.490124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b798ddc6c-jlkxl" event={"ID":"ed605944-4582-4d0f-96cc-a073aa8d99ef","Type":"ContainerStarted","Data":"b6e8607d18bab92c204587b0fcdc1399e62ac2e9a07d0f63e5939f3aff4dad31"} Jul 13 19:35:56.507297 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:56.507251 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b798ddc6c-jlkxl" podStartSLOduration=1.5072349329999999 podStartE2EDuration="1.507234933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:35:56.506040051 +0000 UTC m=+648.067438120" watchObservedRunningTime="2026-07-13 19:35:56.507234933 +0000 UTC m=+648.068632977" Jul 13 19:35:59.130824 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.130789 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd"] Jul 13 19:35:59.134391 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.134374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:35:59.137045 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.137025 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 19:35:59.137326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.137308 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 19:35:59.138163 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.138148 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-r25x8\"" Jul 13 19:35:59.153486 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.153459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd"] Jul 13 19:35:59.224827 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.224791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zhk\" (UniqueName: \"kubernetes.io/projected/e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61-kube-api-access-c8zhk\") pod \"authorino-operator-75cc8d5cb7-pc5dd\" (UID: \"e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:35:59.326285 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.326246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zhk\" (UniqueName: \"kubernetes.io/projected/e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61-kube-api-access-c8zhk\") pod \"authorino-operator-75cc8d5cb7-pc5dd\" (UID: \"e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:35:59.334831 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.334800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zhk\" (UniqueName: \"kubernetes.io/projected/e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61-kube-api-access-c8zhk\") pod \"authorino-operator-75cc8d5cb7-pc5dd\" (UID: \"e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:35:59.450554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.450459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:35:59.584346 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:35:59.584323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd"] Jul 13 19:35:59.587055 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:35:59.587025 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49b46c1_2fcb_4d9d_b7d1_4a59a3eaeb61.slice/crio-54d72dcb2dfab004055e0d1dc4cff59ef30a96f7161d18088ed5bacf4e151595 WatchSource:0}: Error finding container 54d72dcb2dfab004055e0d1dc4cff59ef30a96f7161d18088ed5bacf4e151595: Status 404 returned error can't find the container with id 54d72dcb2dfab004055e0d1dc4cff59ef30a96f7161d18088ed5bacf4e151595 Jul 13 19:36:00.519718 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:00.519679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" event={"ID":"e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61","Type":"ContainerStarted","Data":"54d72dcb2dfab004055e0d1dc4cff59ef30a96f7161d18088ed5bacf4e151595"} Jul 13 19:36:02.532277 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:02.532196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" event={"ID":"e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61","Type":"ContainerStarted","Data":"5d598a99f3e6cfe3fef09205e7264e1fa21e7707b8711291ab294da8df47777a"} Jul 13 19:36:02.532277 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:02.532257 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:36:02.581972 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:02.581902 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" podStartSLOduration=1.035452143 podStartE2EDuration="3.581881894s" totalImagesPullingTime="2.546429751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:36:02.579509569 +0000 UTC m=+654.140907636" watchObservedRunningTime="2026-07-13 19:36:02.581881894 +0000 UTC m=+654.143279935" Jul 13 19:36:05.590510 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:05.590472 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:36:05.590981 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:05.590547 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:36:05.595516 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:05.595489 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:36:06.556243 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:06.556213 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b798ddc6c-jlkxl" Jul 13 19:36:06.626541 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:06.626504 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:36:08.150021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.149989 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp"] Jul 13 19:36:08.152578 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.152560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:08.155057 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.155038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wqmhm\"" Jul 13 19:36:08.163554 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.163532 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp"] Jul 13 19:36:08.309622 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.309586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xkq\" (UniqueName: \"kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq\") pod \"limitador-operator-controller-manager-859d7b78bd-b4skp\" (UID: \"ed699d37-3de2-4439-99b9-a6fe150e7984\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:08.410643 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.410560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xkq\" (UniqueName: \"kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq\") pod \"limitador-operator-controller-manager-859d7b78bd-b4skp\" (UID: \"ed699d37-3de2-4439-99b9-a6fe150e7984\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:08.419949 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.419915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xkq\" (UniqueName: \"kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq\") pod \"limitador-operator-controller-manager-859d7b78bd-b4skp\" (UID: \"ed699d37-3de2-4439-99b9-a6fe150e7984\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:08.462741 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.462704 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:08.598193 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:08.598167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp"] Jul 13 19:36:08.599339 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:36:08.599314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded699d37_3de2_4439_99b9_a6fe150e7984.slice/crio-e79e4cd0154e0dac000be00e0c5f087199f19002c11b77d5b9e76843000fd5e1 WatchSource:0}: Error finding container e79e4cd0154e0dac000be00e0c5f087199f19002c11b77d5b9e76843000fd5e1: Status 404 returned error can't find the container with id e79e4cd0154e0dac000be00e0c5f087199f19002c11b77d5b9e76843000fd5e1 Jul 13 19:36:09.570539 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:09.570493 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" event={"ID":"ed699d37-3de2-4439-99b9-a6fe150e7984","Type":"ContainerStarted","Data":"e79e4cd0154e0dac000be00e0c5f087199f19002c11b77d5b9e76843000fd5e1"} Jul 13 19:36:11.582389 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:11.582354 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" event={"ID":"ed699d37-3de2-4439-99b9-a6fe150e7984","Type":"ContainerStarted","Data":"d34759b1528af57d690d9ed832a786417b0aba60137c833a81a5c765747e18fa"} Jul 13 19:36:11.582822 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:11.582470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:11.601268 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:11.601222 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" podStartSLOduration=1.627231179 podStartE2EDuration="3.601205258s" totalImagesPullingTime="1.973974079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:36:11.597286979 +0000 UTC m=+663.158685037" watchObservedRunningTime="2026-07-13 19:36:11.601205258 +0000 UTC m=+663.162603301" Jul 13 19:36:13.539807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:13.539777 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pc5dd" Jul 13 19:36:22.589720 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:22.589683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:27.538675 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.538642 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp"] Jul 13 19:36:27.539135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.538882 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" containerName="manager" containerID="cri-o://d34759b1528af57d690d9ed832a786417b0aba60137c833a81a5c765747e18fa" gracePeriod=2 Jul 13 19:36:27.550489 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.550465 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp"] Jul 13 19:36:27.565737 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.565711 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4"] Jul 13 19:36:27.566092 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.566081 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" containerName="manager" Jul 13 19:36:27.566135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.566094 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" containerName="manager" Jul 13 19:36:27.566173 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.566150 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" containerName="manager" Jul 13 19:36:27.569244 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.569225 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:27.579173 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.578998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87rd\" (UniqueName: \"kubernetes.io/projected/58d510ca-ca82-473f-9e4d-2104460feca9-kube-api-access-l87rd\") pod \"limitador-operator-controller-manager-859d7b78bd-hwwm4\" (UID: \"58d510ca-ca82-473f-9e4d-2104460feca9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:27.582583 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.582563 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4"] Jul 13 19:36:27.598677 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:27.598645 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:27.666465 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.666403 2582 generic.go:356] "Generic (PLEG): container finished" podID="ed699d37-3de2-4439-99b9-a6fe150e7984" containerID="d34759b1528af57d690d9ed832a786417b0aba60137c833a81a5c765747e18fa" exitCode=0 Jul 13 19:36:27.679926 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.679902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87rd\" (UniqueName: \"kubernetes.io/projected/58d510ca-ca82-473f-9e4d-2104460feca9-kube-api-access-l87rd\") pod \"limitador-operator-controller-manager-859d7b78bd-hwwm4\" (UID: \"58d510ca-ca82-473f-9e4d-2104460feca9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:27.688113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.688086 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87rd\" (UniqueName: \"kubernetes.io/projected/58d510ca-ca82-473f-9e4d-2104460feca9-kube-api-access-l87rd\") pod \"limitador-operator-controller-manager-859d7b78bd-hwwm4\" (UID: \"58d510ca-ca82-473f-9e4d-2104460feca9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:27.788491 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.788463 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:27.790588 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:27.790546 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:27.881120 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.881086 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2xkq\" (UniqueName: \"kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq\") pod \"ed699d37-3de2-4439-99b9-a6fe150e7984\" (UID: \"ed699d37-3de2-4439-99b9-a6fe150e7984\") " Jul 13 19:36:27.883171 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.883146 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq" (OuterVolumeSpecName: "kube-api-access-k2xkq") pod "ed699d37-3de2-4439-99b9-a6fe150e7984" (UID: "ed699d37-3de2-4439-99b9-a6fe150e7984"). InnerVolumeSpecName "kube-api-access-k2xkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:36:27.925352 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.925327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:27.982403 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:27.982351 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xkq\" (UniqueName: \"kubernetes.io/projected/ed699d37-3de2-4439-99b9-a6fe150e7984-kube-api-access-k2xkq\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:28.062654 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.062626 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4"] Jul 13 19:36:28.064657 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:36:28.064627 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d510ca_ca82_473f_9e4d_2104460feca9.slice/crio-37573292dac24f5fe827d4c886c70e2c1f321ce2adb1b8c6e7442d57b271c03e WatchSource:0}: Error finding container 37573292dac24f5fe827d4c886c70e2c1f321ce2adb1b8c6e7442d57b271c03e: Status 404 returned error can't find the container with id 37573292dac24f5fe827d4c886c70e2c1f321ce2adb1b8c6e7442d57b271c03e Jul 13 19:36:28.673101 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.673061 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:28.673590 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.673100 2582 scope.go:117] "RemoveContainer" containerID="d34759b1528af57d690d9ed832a786417b0aba60137c833a81a5c765747e18fa" Jul 13 19:36:28.674965 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.674940 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" event={"ID":"58d510ca-ca82-473f-9e4d-2104460feca9","Type":"ContainerStarted","Data":"d862573f9e6814624a81f3e179d751528da51f27e8205598d60fcb35cc0cc9f5"} Jul 13 19:36:28.675073 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.675014 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" event={"ID":"58d510ca-ca82-473f-9e4d-2104460feca9","Type":"ContainerStarted","Data":"37573292dac24f5fe827d4c886c70e2c1f321ce2adb1b8c6e7442d57b271c03e"} Jul 13 19:36:28.675073 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.675045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:36:28.675569 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:28.675546 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:28.692829 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.692782 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" podStartSLOduration=1.6927687040000001 podStartE2EDuration="1.692768704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:36:28.691284181 +0000 UTC m=+680.252682246" watchObservedRunningTime="2026-07-13 19:36:28.692768704 +0000 UTC m=+680.254166745" Jul 13 19:36:28.693094 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:28.693058 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:28.694719 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:28.694697 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:28.960452 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:28.960356 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-b4skp\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b4skp" Jul 13 19:36:28.961569 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:28.961550 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed699d37-3de2-4439-99b9-a6fe150e7984" path="/var/lib/kubelet/pods/ed699d37-3de2-4439-99b9-a6fe150e7984/volumes" Jul 13 19:36:31.649853 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.649808 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d6f7ccd87-ghxwd" podUID="0201c78a-91eb-4799-881f-d350e17cdc61" containerName="console" containerID="cri-o://d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f" gracePeriod=15 Jul 13 19:36:31.906720 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.906664 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:36:31.909810 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909787 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.909930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909826 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.909930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909844 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6tbg\" (UniqueName: \"kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.909930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909873 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.909930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909915 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.910135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.909938 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.910135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.910004 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert\") pod \"0201c78a-91eb-4799-881f-d350e17cdc61\" (UID: \"0201c78a-91eb-4799-881f-d350e17cdc61\") " Jul 13 19:36:31.910395 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.910320 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca" (OuterVolumeSpecName: "service-ca") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:36:31.910395 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.910324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:36:31.910598 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.910442 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:36:31.910598 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.910539 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config" (OuterVolumeSpecName: "console-config") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:36:31.912079 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.912057 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:36:31.912223 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.912200 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:36:31.912269 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:31.912258 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg" (OuterVolumeSpecName: "kube-api-access-f6tbg") pod "0201c78a-91eb-4799-881f-d350e17cdc61" (UID: "0201c78a-91eb-4799-881f-d350e17cdc61"). InnerVolumeSpecName "kube-api-access-f6tbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:36:32.011046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011013 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011040 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-oauth-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011046 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011052 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-trusted-ca-bundle\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011065 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-serving-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011077 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0201c78a-91eb-4799-881f-d350e17cdc61-console-oauth-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011097 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0201c78a-91eb-4799-881f-d350e17cdc61-console-config\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.011289 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.011109 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6tbg\" (UniqueName: \"kubernetes.io/projected/0201c78a-91eb-4799-881f-d350e17cdc61-kube-api-access-f6tbg\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:36:32.700676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.700641 2582 generic.go:356] "Generic (PLEG): container finished" podID="0201c78a-91eb-4799-881f-d350e17cdc61" containerID="d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f" exitCode=0 Jul 13 19:36:32.701153 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.700685 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6f7ccd87-ghxwd" Jul 13 19:36:32.701153 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.700731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6f7ccd87-ghxwd" event={"ID":"0201c78a-91eb-4799-881f-d350e17cdc61","Type":"ContainerDied","Data":"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f"} Jul 13 19:36:32.701153 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.700788 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6f7ccd87-ghxwd" event={"ID":"0201c78a-91eb-4799-881f-d350e17cdc61","Type":"ContainerDied","Data":"ac01f88b183800d32305bf8b9c9a2eff9be4764bb1e1341ab1bbb0bdc9e800e4"} Jul 13 19:36:32.701153 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.700822 2582 scope.go:117] "RemoveContainer" containerID="d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f" Jul 13 19:36:32.710326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.710311 2582 scope.go:117] "RemoveContainer" containerID="d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f" Jul 13 19:36:32.710600 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:36:32.710584 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f\": container with ID starting with d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f not found: ID does not exist" containerID="d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f" Jul 13 19:36:32.710650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.710612 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f"} err="failed to get container status \"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f\": rpc error: code = NotFound desc = could not find container \"d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f\": container with ID starting with d38df830003b60a5a9a23cc107790594a298540337352c51878a2a634f0c066f not found: ID does not exist" Jul 13 19:36:32.722538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.722514 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:36:32.729027 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.729008 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d6f7ccd87-ghxwd"] Jul 13 19:36:32.961348 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:32.961275 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0201c78a-91eb-4799-881f-d350e17cdc61" path="/var/lib/kubelet/pods/0201c78a-91eb-4799-881f-d350e17cdc61/volumes" Jul 13 19:36:39.686381 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:36:39.686351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwwm4" Jul 13 19:37:48.579142 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.579112 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:37:48.579638 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.579543 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201c78a-91eb-4799-881f-d350e17cdc61" containerName="console" Jul 13 19:37:48.579638 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.579555 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201c78a-91eb-4799-881f-d350e17cdc61" containerName="console" Jul 13 19:37:48.579638 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.579611 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0201c78a-91eb-4799-881f-d350e17cdc61" containerName="console" Jul 13 19:37:48.582464 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.582448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:48.584963 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.584941 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 19:37:48.585198 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.585182 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 19:37:48.585299 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.585221 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4ppkp\"" Jul 13 19:37:48.592237 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.592214 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:37:48.647808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.647770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrvt\" (UniqueName: \"kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:48.647982 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.647827 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:48.748856 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.748818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrvt\" (UniqueName: \"kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:48.749038 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.748875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:48.749038 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:37:48.748999 2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 13 19:37:48.749129 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:37:48.749052 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert podName:0badeeb4-82af-4876-bdc6-8d296d4fb3e2 nodeName:}" failed. No retries permitted until 2026-07-13 19:37:49.249037138 +0000 UTC m=+760.810435160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert") pod "maas-controller-65fb85bbdb-gt4th" (UID: "0badeeb4-82af-4876-bdc6-8d296d4fb3e2") : secret "maas-controller-webhook-cert" not found Jul 13 19:37:48.759725 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:48.759696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrvt\" (UniqueName: \"kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:49.253312 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:49.253273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:49.255884 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:49.255861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") pod \"maas-controller-65fb85bbdb-gt4th\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:49.494543 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:49.494506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:49.633764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:49.633736 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:37:49.635578 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:37:49.635556 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0badeeb4_82af_4876_bdc6_8d296d4fb3e2.slice/crio-a920b40a4fe1c1653b59d9acb242e04ffd2b65326897cb4e90c464ad1865b16a WatchSource:0}: Error finding container a920b40a4fe1c1653b59d9acb242e04ffd2b65326897cb4e90c464ad1865b16a: Status 404 returned error can't find the container with id a920b40a4fe1c1653b59d9acb242e04ffd2b65326897cb4e90c464ad1865b16a Jul 13 19:37:50.119764 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:50.119726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" event={"ID":"0badeeb4-82af-4876-bdc6-8d296d4fb3e2","Type":"ContainerStarted","Data":"a920b40a4fe1c1653b59d9acb242e04ffd2b65326897cb4e90c464ad1865b16a"} Jul 13 19:37:52.135068 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:52.135034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" event={"ID":"0badeeb4-82af-4876-bdc6-8d296d4fb3e2","Type":"ContainerStarted","Data":"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb"} Jul 13 19:37:52.135538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:52.135095 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:37:52.152942 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:37:52.152883 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" podStartSLOduration=2.242273363 podStartE2EDuration="4.152868894s" totalImagesPullingTime="1.910595531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:37:52.151750703 +0000 UTC m=+763.713148764" watchObservedRunningTime="2026-07-13 19:37:52.152868894 +0000 UTC m=+763.714266937" Jul 13 19:38:03.142634 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:03.142606 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:38:25.377791 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.377755 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7"] Jul 13 19:38:25.381555 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.381534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.384037 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.384014 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 19:38:25.384170 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.384058 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gppll\"" Jul 13 19:38:25.394326 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.394294 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7"] Jul 13 19:38:25.468362 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.468328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead7469-242a-4086-9e62-f2432fcf3633-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.468362 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.468369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns26m\" (UniqueName: \"kubernetes.io/projected/3ead7469-242a-4086-9e62-f2432fcf3633-kube-api-access-ns26m\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.569220 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.569188 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead7469-242a-4086-9e62-f2432fcf3633-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.569392 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.569231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns26m\" (UniqueName: \"kubernetes.io/projected/3ead7469-242a-4086-9e62-f2432fcf3633-kube-api-access-ns26m\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.569921 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.569900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead7469-242a-4086-9e62-f2432fcf3633-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.577850 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.577807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns26m\" (UniqueName: \"kubernetes.io/projected/3ead7469-242a-4086-9e62-f2432fcf3633-kube-api-access-ns26m\") pod \"payload-pre-processing-74584bb9f8-vdqx7\" (UID: \"3ead7469-242a-4086-9e62-f2432fcf3633\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.693216 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.693135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:25.843547 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:25.843519 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7"] Jul 13 19:38:25.846088 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:38:25.846055 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ead7469_242a_4086_9e62_f2432fcf3633.slice/crio-90b9d944c29df86e89fb477a738ad3f5c825731f12f4f85556d2c0fbee7251e0 WatchSource:0}: Error finding container 90b9d944c29df86e89fb477a738ad3f5c825731f12f4f85556d2c0fbee7251e0: Status 404 returned error can't find the container with id 90b9d944c29df86e89fb477a738ad3f5c825731f12f4f85556d2c0fbee7251e0 Jul 13 19:38:26.338190 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:26.338154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" event={"ID":"3ead7469-242a-4086-9e62-f2432fcf3633","Type":"ContainerStarted","Data":"90b9d944c29df86e89fb477a738ad3f5c825731f12f4f85556d2c0fbee7251e0"} Jul 13 19:38:27.345307 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:27.345271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" event={"ID":"3ead7469-242a-4086-9e62-f2432fcf3633","Type":"ContainerStarted","Data":"672edd78b1347d5d1a15d03651ff142f4369c593354f463e17d7c1000b41d3de"} Jul 13 19:38:27.345788 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:27.345389 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:27.363471 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:27.363408 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" podStartSLOduration=0.965734276 podStartE2EDuration="2.363392096s" totalImagesPullingTime="1.39765782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:38:27.361378716 +0000 UTC m=+798.922776762" watchObservedRunningTime="2026-07-13 19:38:27.363392096 +0000 UTC m=+798.924790139" Jul 13 19:38:38.354139 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:38.354064 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-vdqx7" Jul 13 19:38:43.367662 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.367632 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb"] Jul 13 19:38:43.370845 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.370822 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.373308 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.373287 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 19:38:43.374251 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.374229 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rx4h8\"" Jul 13 19:38:43.374373 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.374272 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 19:38:43.374373 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.374275 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 13 19:38:43.380007 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.379984 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb"] Jul 13 19:38:43.412722 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.412844 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.412844 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxfz\" (UniqueName: \"kubernetes.io/projected/d577c41d-482b-4188-b73d-2bb28157753f-kube-api-access-7hxfz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.412930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412856 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577c41d-482b-4188-b73d-2bb28157753f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.412930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.412930 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.412919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.513859 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577c41d-482b-4188-b73d-2bb28157753f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.513859 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514057 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514057 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514057 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514057 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.513970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxfz\" (UniqueName: \"kubernetes.io/projected/d577c41d-482b-4188-b73d-2bb28157753f-kube-api-access-7hxfz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514441 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.514409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514586 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.514455 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.514586 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.514463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.516349 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.516321 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577c41d-482b-4188-b73d-2bb28157753f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.516604 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.516586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577c41d-482b-4188-b73d-2bb28157753f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.521696 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.521673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxfz\" (UniqueName: \"kubernetes.io/projected/d577c41d-482b-4188-b73d-2bb28157753f-kube-api-access-7hxfz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb\" (UID: \"d577c41d-482b-4188-b73d-2bb28157753f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.682473 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.682371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:43.827523 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:43.827492 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb"] Jul 13 19:38:43.828969 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:38:43.828938 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd577c41d_482b_4188_b73d_2bb28157753f.slice/crio-93f19947aab02a27021e4615398ca28a499f962e2f52ff7bd0951d5b01309bb9 WatchSource:0}: Error finding container 93f19947aab02a27021e4615398ca28a499f962e2f52ff7bd0951d5b01309bb9: Status 404 returned error can't find the container with id 93f19947aab02a27021e4615398ca28a499f962e2f52ff7bd0951d5b01309bb9 Jul 13 19:38:44.447751 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:44.447703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" event={"ID":"d577c41d-482b-4188-b73d-2bb28157753f","Type":"ContainerStarted","Data":"93f19947aab02a27021e4615398ca28a499f962e2f52ff7bd0951d5b01309bb9"} Jul 13 19:38:45.454762 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:45.454722 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" event={"ID":"d577c41d-482b-4188-b73d-2bb28157753f","Type":"ContainerStarted","Data":"3fe102c4c766f2582c0fdfd10c937eb9ad2ba51f8224bcd967486685c43a463e"} Jul 13 19:38:45.474063 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:45.474019 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" podStartSLOduration=1.4298888060000001 podStartE2EDuration="2.47400357s" totalImagesPullingTime="1.044114764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:38:45.471128399 +0000 UTC m=+817.032526472" watchObservedRunningTime="2026-07-13 19:38:45.47400357 +0000 UTC m=+817.035401613" Jul 13 19:38:53.682514 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:53.682477 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:53.682917 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:53.682561 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:53.694826 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:53.694802 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:38:54.514225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:38:54.514197 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb" Jul 13 19:39:07.783477 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.783438 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6"] Jul 13 19:39:07.787377 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.787351 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.789873 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.789848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 13 19:39:07.800048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.800021 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6"] Jul 13 19:39:07.931621 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.931807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.931807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk42q\" (UniqueName: \"kubernetes.io/projected/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-kube-api-access-tk42q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.931807 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.931988 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:07.931988 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:07.931915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.032941 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.032906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033127 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.032966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033127 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033127 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033127 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033127 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk42q\" (UniqueName: \"kubernetes.io/projected/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-kube-api-access-tk42q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033406 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033372 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033406 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.033592 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.033548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.035404 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.035385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.035624 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.035606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.041004 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.040982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk42q\" (UniqueName: \"kubernetes.io/projected/f0597483-a872-4ae4-a35d-f63b3d5d1cdd-kube-api-access-tk42q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6\" (UID: \"f0597483-a872-4ae4-a35d-f63b3d5d1cdd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.099129 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.099096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:08.233889 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.233856 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6"] Jul 13 19:39:08.234776 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:39:08.234747 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0597483_a872_4ae4_a35d_f63b3d5d1cdd.slice/crio-bd0225c91dd58f75b11f96439b54f4a950b1dccaa10e98c3afad04360b034dde WatchSource:0}: Error finding container bd0225c91dd58f75b11f96439b54f4a950b1dccaa10e98c3afad04360b034dde: Status 404 returned error can't find the container with id bd0225c91dd58f75b11f96439b54f4a950b1dccaa10e98c3afad04360b034dde Jul 13 19:39:08.579435 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.579381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" event={"ID":"f0597483-a872-4ae4-a35d-f63b3d5d1cdd","Type":"ContainerStarted","Data":"8b4c876e6453b86bea61733e19f556adcb493ea91c0c57df5dea6db5edee8326"} Jul 13 19:39:08.579626 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.579491 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" event={"ID":"f0597483-a872-4ae4-a35d-f63b3d5d1cdd","Type":"ContainerStarted","Data":"bd0225c91dd58f75b11f96439b54f4a950b1dccaa10e98c3afad04360b034dde"} Jul 13 19:39:08.599828 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:08.599785 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" podStartSLOduration=1.4092530779999999 podStartE2EDuration="1.599771743s" totalImagesPullingTime="190.518665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:39:08.59767229 +0000 UTC m=+840.159070376" watchObservedRunningTime="2026-07-13 19:39:08.599771743 +0000 UTC m=+840.161169787" Jul 13 19:39:18.099489 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:18.099440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:18.099913 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:18.099511 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:18.112492 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:18.112462 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:39:18.643541 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:39:18.643512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6" Jul 13 19:40:08.911339 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:40:08.911311 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:40:08.912282 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:40:08.912259 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:42:08.516921 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.516881 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:42:08.517500 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.517179 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" podUID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" containerName="manager" containerID="cri-o://c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb" gracePeriod=30 Jul 13 19:42:08.774700 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.774639 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:42:08.820134 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.820104 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtrvt\" (UniqueName: \"kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt\") pod \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " Jul 13 19:42:08.820337 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.820175 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") pod \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\" (UID: \"0badeeb4-82af-4876-bdc6-8d296d4fb3e2\") " Jul 13 19:42:08.822397 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.822368 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0badeeb4-82af-4876-bdc6-8d296d4fb3e2" (UID: "0badeeb4-82af-4876-bdc6-8d296d4fb3e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:42:08.822537 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.822404 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt" (OuterVolumeSpecName: "kube-api-access-xtrvt") pod "0badeeb4-82af-4876-bdc6-8d296d4fb3e2" (UID: "0badeeb4-82af-4876-bdc6-8d296d4fb3e2"). InnerVolumeSpecName "kube-api-access-xtrvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:42:08.921667 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.921633 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrvt\" (UniqueName: \"kubernetes.io/projected/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-kube-api-access-xtrvt\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:42:08.921667 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:08.921662 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0badeeb4-82af-4876-bdc6-8d296d4fb3e2-webhook-cert\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:42:09.552670 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.552632 2582 generic.go:356] "Generic (PLEG): container finished" podID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" containerID="c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb" exitCode=0 Jul 13 19:42:09.553113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.552695 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" Jul 13 19:42:09.553113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.552720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" event={"ID":"0badeeb4-82af-4876-bdc6-8d296d4fb3e2","Type":"ContainerDied","Data":"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb"} Jul 13 19:42:09.553113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.552776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65fb85bbdb-gt4th" event={"ID":"0badeeb4-82af-4876-bdc6-8d296d4fb3e2","Type":"ContainerDied","Data":"a920b40a4fe1c1653b59d9acb242e04ffd2b65326897cb4e90c464ad1865b16a"} Jul 13 19:42:09.553113 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.552799 2582 scope.go:117] "RemoveContainer" containerID="c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb" Jul 13 19:42:09.561951 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.561934 2582 scope.go:117] "RemoveContainer" containerID="c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb" Jul 13 19:42:09.562196 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:42:09.562177 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb\": container with ID starting with c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb not found: ID does not exist" containerID="c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb" Jul 13 19:42:09.562268 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.562202 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb"} err="failed to get container status \"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb\": rpc error: code = NotFound desc = could not find container \"c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb\": container with ID starting with c57cd7de9e47e3d1e5af553f2a8c1d79152d164d216ea1ebd1e4c338e44730bb not found: ID does not exist" Jul 13 19:42:09.569404 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.569385 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:42:09.573091 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:09.573067 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65fb85bbdb-gt4th"] Jul 13 19:42:10.961184 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:42:10.961151 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" path="/var/lib/kubelet/pods/0badeeb4-82af-4876-bdc6-8d296d4fb3e2/volumes" Jul 13 19:45:08.946876 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:45:08.946844 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:45:08.949313 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:45:08.949291 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:50:08.986267 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:50:08.986236 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:50:08.989411 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:50:08.989394 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:54:30.690195 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.690157 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g"] Jul 13 19:54:30.690645 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.690577 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" containerName="manager" Jul 13 19:54:30.690645 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.690589 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" containerName="manager" Jul 13 19:54:30.690645 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.690646 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0badeeb4-82af-4876-bdc6-8d296d4fb3e2" containerName="manager" Jul 13 19:54:30.693740 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.693725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:30.697453 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.697104 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 19:54:30.697453 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.697104 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 19:54:30.699580 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.699554 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g"] Jul 13 19:54:30.817232 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.817183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g\" (UID: \"9432562d-02f7-496a-8d2c-3e1f289e96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:30.918508 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.918479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g\" (UID: \"9432562d-02f7-496a-8d2c-3e1f289e96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:30.919149 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:30.919127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g\" (UID: \"9432562d-02f7-496a-8d2c-3e1f289e96eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:31.005562 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:31.005481 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:31.346923 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:31.346891 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g"] Jul 13 19:54:31.348761 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:54:31.348736 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9432562d_02f7_496a_8d2c_3e1f289e96eb.slice/crio-3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0 WatchSource:0}: Error finding container 3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0: Status 404 returned error can't find the container with id 3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0 Jul 13 19:54:31.350734 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:31.350716 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 19:54:31.456689 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:31.456654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" event={"ID":"9432562d-02f7-496a-8d2c-3e1f289e96eb","Type":"ContainerStarted","Data":"3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0"} Jul 13 19:54:37.490650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:37.490552 2582 generic.go:356] "Generic (PLEG): container finished" podID="9432562d-02f7-496a-8d2c-3e1f289e96eb" containerID="58e95c2926713f0e6c2c2e5aba4334b90c73983b912703f2b69c7a1a9f298ae8" exitCode=0 Jul 13 19:54:37.490650 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:37.490638 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" event={"ID":"9432562d-02f7-496a-8d2c-3e1f289e96eb","Type":"ContainerDied","Data":"58e95c2926713f0e6c2c2e5aba4334b90c73983b912703f2b69c7a1a9f298ae8"} Jul 13 19:54:38.634126 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:38.634097 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:38.672347 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:38.672317 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca\") pod \"9432562d-02f7-496a-8d2c-3e1f289e96eb\" (UID: \"9432562d-02f7-496a-8d2c-3e1f289e96eb\") " Jul 13 19:54:38.672693 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:38.672666 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9432562d-02f7-496a-8d2c-3e1f289e96eb" (UID: "9432562d-02f7-496a-8d2c-3e1f289e96eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:54:38.773659 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:38.773573 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9432562d-02f7-496a-8d2c-3e1f289e96eb-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:54:39.503100 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:39.503055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" event={"ID":"9432562d-02f7-496a-8d2c-3e1f289e96eb","Type":"ContainerDied","Data":"3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0"} Jul 13 19:54:39.503100 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:39.503101 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g" Jul 13 19:54:39.503294 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:54:39.503138 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7a2ebedde9f3a4be7e32fd2476f2542f8412a4dda91b3d318ba9f15a6676b0" Jul 13 19:55:09.028895 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:55:09.028868 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:55:09.033710 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:55:09.033678 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 19:56:10.068404 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.068364 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:56:10.068932 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.068804 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9432562d-02f7-496a-8d2c-3e1f289e96eb" containerName="revoke-keys" Jul 13 19:56:10.068932 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.068816 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9432562d-02f7-496a-8d2c-3e1f289e96eb" containerName="revoke-keys" Jul 13 19:56:10.068932 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.068876 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9432562d-02f7-496a-8d2c-3e1f289e96eb" containerName="revoke-keys" Jul 13 19:56:10.071808 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.071788 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.074506 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.074485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-facb3ea1-serving-cert\"" Jul 13 19:56:10.074696 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.074677 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 19:56:10.075741 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.075684 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 19:56:10.075741 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.075730 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 19:56:10.083318 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.083299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:56:10.158816 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.158785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skq6\" (UniqueName: \"kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.158985 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.158854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.158985 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.158878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.259820 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.259781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skq6\" (UniqueName: \"kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.259982 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.259856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.259982 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.259879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.260057 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:56:10.260000 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-serving-cert: secret "maas-api-e2e-ait-facb3ea1-serving-cert" not found Jul 13 19:56:10.260099 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:56:10.260072 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls podName:2f5100de-e382-4a24-9cf7-fd560e9386bb nodeName:}" failed. No retries permitted until 2026-07-13 19:56:10.760055864 +0000 UTC m=+1862.321453889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls") pod "maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" (UID: "2f5100de-e382-4a24-9cf7-fd560e9386bb") : secret "maas-api-e2e-ait-facb3ea1-serving-cert" not found Jul 13 19:56:10.260472 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.260453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.269411 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.269388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skq6\" (UniqueName: \"kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.766172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.766141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.768743 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.768720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") pod \"maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:10.984256 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:10.984222 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:11.119203 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:11.119172 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:56:11.120489 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:56:11.120468 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5100de_e382_4a24_9cf7_fd560e9386bb.slice/crio-04054202c1a1c68d61643d4d56cb92b2259776b9171405ef210976a71973cb15 WatchSource:0}: Error finding container 04054202c1a1c68d61643d4d56cb92b2259776b9171405ef210976a71973cb15: Status 404 returned error can't find the container with id 04054202c1a1c68d61643d4d56cb92b2259776b9171405ef210976a71973cb15 Jul 13 19:56:12.005666 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:12.005634 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" event={"ID":"2f5100de-e382-4a24-9cf7-fd560e9386bb","Type":"ContainerStarted","Data":"04054202c1a1c68d61643d4d56cb92b2259776b9171405ef210976a71973cb15"} Jul 13 19:56:14.019821 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:14.019775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" event={"ID":"2f5100de-e382-4a24-9cf7-fd560e9386bb","Type":"ContainerStarted","Data":"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e"} Jul 13 19:56:14.020232 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:14.019863 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:56:14.037665 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:14.037606 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" podStartSLOduration=2.074445856 podStartE2EDuration="4.037587583s" totalImagesPullingTime="1.963141727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:56:14.036510355 +0000 UTC m=+1865.597908418" watchObservedRunningTime="2026-07-13 19:56:14.037587583 +0000 UTC m=+1865.598985631" Jul 13 19:56:20.030641 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:56:20.030614 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:57:15.180377 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.180331 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw"] Jul 13 19:57:15.184000 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.183982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:15.186605 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.186580 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 19:57:15.191135 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.191110 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw"] Jul 13 19:57:15.261873 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.261828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw\" (UID: \"3269aabc-cf9b-42ab-af4f-2cefc83dac25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:15.363031 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.362993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw\" (UID: \"3269aabc-cf9b-42ab-af4f-2cefc83dac25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:15.363656 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.363635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw\" (UID: \"3269aabc-cf9b-42ab-af4f-2cefc83dac25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:15.496309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.496220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:15.639664 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:15.639622 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw"] Jul 13 19:57:15.639930 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:57:15.639908 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3269aabc_cf9b_42ab_af4f_2cefc83dac25.slice/crio-ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b WatchSource:0}: Error finding container ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b: Status 404 returned error can't find the container with id ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b Jul 13 19:57:16.362217 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:16.362182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" event={"ID":"3269aabc-cf9b-42ab-af4f-2cefc83dac25","Type":"ContainerStarted","Data":"d590dc1615a45afeb5a145ec18c3eae31764dd526feb05d88c73d5134a7926f7"} Jul 13 19:57:16.362629 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:16.362234 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" event={"ID":"3269aabc-cf9b-42ab-af4f-2cefc83dac25","Type":"ContainerStarted","Data":"ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b"} Jul 13 19:57:16.378399 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:16.378354 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" podStartSLOduration=1.378330496 podStartE2EDuration="1.378330496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:57:16.377510101 +0000 UTC m=+1927.938908146" watchObservedRunningTime="2026-07-13 19:57:16.378330496 +0000 UTC m=+1927.939728573" Jul 13 19:57:17.369750 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:17.369713 2582 generic.go:356] "Generic (PLEG): container finished" podID="3269aabc-cf9b-42ab-af4f-2cefc83dac25" containerID="d590dc1615a45afeb5a145ec18c3eae31764dd526feb05d88c73d5134a7926f7" exitCode=0 Jul 13 19:57:17.370155 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:17.369805 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" event={"ID":"3269aabc-cf9b-42ab-af4f-2cefc83dac25","Type":"ContainerDied","Data":"d590dc1615a45afeb5a145ec18c3eae31764dd526feb05d88c73d5134a7926f7"} Jul 13 19:57:18.518245 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:18.518216 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:18.585955 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:18.585926 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca\") pod \"3269aabc-cf9b-42ab-af4f-2cefc83dac25\" (UID: \"3269aabc-cf9b-42ab-af4f-2cefc83dac25\") " Jul 13 19:57:18.586269 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:18.586246 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3269aabc-cf9b-42ab-af4f-2cefc83dac25" (UID: "3269aabc-cf9b-42ab-af4f-2cefc83dac25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:57:18.686648 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:18.686548 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3269aabc-cf9b-42ab-af4f-2cefc83dac25-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:57:19.381935 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:19.381908 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" Jul 13 19:57:19.382116 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:19.381899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw" event={"ID":"3269aabc-cf9b-42ab-af4f-2cefc83dac25","Type":"ContainerDied","Data":"ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b"} Jul 13 19:57:19.382116 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:19.382031 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff87d637fdbcfa1e91819eb568bccb2bb045af762905f321bb7e0dbc9af5b68b" Jul 13 19:57:25.534178 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.534140 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:57:25.534717 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.534461 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" podUID="2f5100de-e382-4a24-9cf7-fd560e9386bb" containerName="maas-api" containerID="cri-o://386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e" gracePeriod=30 Jul 13 19:57:25.788234 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.788172 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:57:25.848686 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.848656 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") pod \"2f5100de-e382-4a24-9cf7-fd560e9386bb\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " Jul 13 19:57:25.848850 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.848733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8skq6\" (UniqueName: \"kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6\") pod \"2f5100de-e382-4a24-9cf7-fd560e9386bb\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " Jul 13 19:57:25.848922 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.848897 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca\") pod \"2f5100de-e382-4a24-9cf7-fd560e9386bb\" (UID: \"2f5100de-e382-4a24-9cf7-fd560e9386bb\") " Jul 13 19:57:25.849225 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.849201 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f5100de-e382-4a24-9cf7-fd560e9386bb" (UID: "2f5100de-e382-4a24-9cf7-fd560e9386bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:57:25.850972 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.850950 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f5100de-e382-4a24-9cf7-fd560e9386bb" (UID: "2f5100de-e382-4a24-9cf7-fd560e9386bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:57:25.851066 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.851017 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6" (OuterVolumeSpecName: "kube-api-access-8skq6") pod "2f5100de-e382-4a24-9cf7-fd560e9386bb" (UID: "2f5100de-e382-4a24-9cf7-fd560e9386bb"). InnerVolumeSpecName "kube-api-access-8skq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:57:25.950696 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.950660 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skq6\" (UniqueName: \"kubernetes.io/projected/2f5100de-e382-4a24-9cf7-fd560e9386bb-kube-api-access-8skq6\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:57:25.950696 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.950690 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5100de-e382-4a24-9cf7-fd560e9386bb-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:57:25.950696 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:25.950701 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5100de-e382-4a24-9cf7-fd560e9386bb-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:57:26.421140 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.421103 2582 generic.go:356] "Generic (PLEG): container finished" podID="2f5100de-e382-4a24-9cf7-fd560e9386bb" containerID="386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e" exitCode=0 Jul 13 19:57:26.421140 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.421142 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" Jul 13 19:57:26.421349 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.421197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" event={"ID":"2f5100de-e382-4a24-9cf7-fd560e9386bb","Type":"ContainerDied","Data":"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e"} Jul 13 19:57:26.421349 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.421248 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl" event={"ID":"2f5100de-e382-4a24-9cf7-fd560e9386bb","Type":"ContainerDied","Data":"04054202c1a1c68d61643d4d56cb92b2259776b9171405ef210976a71973cb15"} Jul 13 19:57:26.421349 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.421275 2582 scope.go:117] "RemoveContainer" containerID="386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e" Jul 13 19:57:26.430953 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.430937 2582 scope.go:117] "RemoveContainer" containerID="386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e" Jul 13 19:57:26.431172 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:57:26.431153 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e\": container with ID starting with 386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e not found: ID does not exist" containerID="386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e" Jul 13 19:57:26.431208 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.431183 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e"} err="failed to get container status \"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e\": rpc error: code = NotFound desc = could not find container \"386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e\": container with ID starting with 386dc7d1b3c227766376f863d8e10a00df5511b96432f19cb76bbac90f6dd61e not found: ID does not exist" Jul 13 19:57:26.442842 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.442819 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:57:26.446467 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.446420 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-facb3ea1-5dd9477bf7-8xshl"] Jul 13 19:57:26.962272 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:26.962237 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5100de-e382-4a24-9cf7-fd560e9386bb" path="/var/lib/kubelet/pods/2f5100de-e382-4a24-9cf7-fd560e9386bb/volumes" Jul 13 19:57:43.086047 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086010 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:57:43.086627 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086610 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3269aabc-cf9b-42ab-af4f-2cefc83dac25" containerName="revoke-keys" Jul 13 19:57:43.086682 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086631 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3269aabc-cf9b-42ab-af4f-2cefc83dac25" containerName="revoke-keys" Jul 13 19:57:43.086682 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086671 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5100de-e382-4a24-9cf7-fd560e9386bb" containerName="maas-api" Jul 13 19:57:43.086682 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086680 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5100de-e382-4a24-9cf7-fd560e9386bb" containerName="maas-api" Jul 13 19:57:43.086790 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086779 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5100de-e382-4a24-9cf7-fd560e9386bb" containerName="maas-api" Jul 13 19:57:43.086826 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.086796 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3269aabc-cf9b-42ab-af4f-2cefc83dac25" containerName="revoke-keys" Jul 13 19:57:43.092591 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.092569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.095172 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.095153 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-66d7f166-serving-cert\"" Jul 13 19:57:43.095300 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.095251 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 19:57:43.096451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.096416 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 19:57:43.096726 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.096411 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 19:57:43.100311 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.100290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:57:43.205264 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.205222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.205451 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.205377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdbr\" (UniqueName: \"kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.205530 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.205508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.306517 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.306476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.306729 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.306534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.306729 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.306617 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdbr\" (UniqueName: \"kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.307208 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.307183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.309064 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.309042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.327915 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.327875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdbr\" (UniqueName: \"kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr\") pod \"maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.406467 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.406408 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:43.555793 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:43.555769 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:57:43.557343 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:57:43.557316 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9599026a_acfb_4367_865e_4061d9f4ad6e.slice/crio-0d44d95ec5ea1f610db40bb483892c06ce560dfbaedb37c65c11d72d9d5b820b WatchSource:0}: Error finding container 0d44d95ec5ea1f610db40bb483892c06ce560dfbaedb37c65c11d72d9d5b820b: Status 404 returned error can't find the container with id 0d44d95ec5ea1f610db40bb483892c06ce560dfbaedb37c65c11d72d9d5b820b Jul 13 19:57:44.524106 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:44.524063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" event={"ID":"9599026a-acfb-4367-865e-4061d9f4ad6e","Type":"ContainerStarted","Data":"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf"} Jul 13 19:57:44.524571 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:44.524159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" event={"ID":"9599026a-acfb-4367-865e-4061d9f4ad6e","Type":"ContainerStarted","Data":"0d44d95ec5ea1f610db40bb483892c06ce560dfbaedb37c65c11d72d9d5b820b"} Jul 13 19:57:44.524571 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:44.524321 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:44.546277 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:44.546219 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" podStartSLOduration=1.073608279 podStartE2EDuration="1.546200326s" totalImagesPullingTime="472.592047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:57:44.545304913 +0000 UTC m=+1956.106702957" watchObservedRunningTime="2026-07-13 19:57:44.546200326 +0000 UTC m=+1956.107598371" Jul 13 19:57:47.460472 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.460435 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:57:47.464309 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.464287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:57:47.466936 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.466913 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 19:57:47.470899 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.470875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:57:47.546610 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.546576 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p\" (UID: \"59b0e682-1252-4db1-813b-7458f0f82c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:57:47.647310 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.647272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p\" (UID: \"59b0e682-1252-4db1-813b-7458f0f82c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:57:47.648001 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.647981 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p\" (UID: \"59b0e682-1252-4db1-813b-7458f0f82c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:57:47.777502 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.777392 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:57:47.932262 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:47.932235 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:57:47.932998 ip-10-0-135-83 kubenswrapper[2582]: W0713 19:57:47.932968 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0e682_1252_4db1_813b_7458f0f82c3b.slice/crio-88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd WatchSource:0}: Error finding container 88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd: Status 404 returned error can't find the container with id 88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd Jul 13 19:57:48.548652 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:48.548626 2582 generic.go:356] "Generic (PLEG): container finished" podID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerID="ceeecdcec5ff4b12ad64278a7889be856abfde149852e4cc731e612b02fd26a4" exitCode=7 Jul 13 19:57:48.548960 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:48.548704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" event={"ID":"59b0e682-1252-4db1-813b-7458f0f82c3b","Type":"ContainerDied","Data":"ceeecdcec5ff4b12ad64278a7889be856abfde149852e4cc731e612b02fd26a4"} Jul 13 19:57:48.548960 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:48.548758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" event={"ID":"59b0e682-1252-4db1-813b-7458f0f82c3b","Type":"ContainerStarted","Data":"88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd"} Jul 13 19:57:48.548960 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:48.548932 2582 scope.go:117] "RemoveContainer" containerID="ceeecdcec5ff4b12ad64278a7889be856abfde149852e4cc731e612b02fd26a4" Jul 13 19:57:49.559117 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:49.559028 2582 generic.go:356] "Generic (PLEG): container finished" podID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerID="a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb" exitCode=7 Jul 13 19:57:49.559540 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:49.559118 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" event={"ID":"59b0e682-1252-4db1-813b-7458f0f82c3b","Type":"ContainerDied","Data":"a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb"} Jul 13 19:57:49.559540 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:49.559187 2582 scope.go:117] "RemoveContainer" containerID="ceeecdcec5ff4b12ad64278a7889be856abfde149852e4cc731e612b02fd26a4" Jul 13 19:57:49.559540 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:49.559460 2582 scope.go:117] "RemoveContainer" containerID="a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb" Jul 13 19:57:49.566144 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:57:49.560061 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p_odh-ai-gateway-infra(59b0e682-1252-4db1-813b-7458f0f82c3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" Jul 13 19:57:50.536381 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:50.536350 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:57:50.568386 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:57:50.568360 2582 scope.go:117] "RemoveContainer" containerID="a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb" Jul 13 19:57:50.568875 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:57:50.568598 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p_odh-ai-gateway-infra(59b0e682-1252-4db1-813b-7458f0f82c3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" Jul 13 19:58:05.955725 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:05.955643 2582 scope.go:117] "RemoveContainer" containerID="a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb" Jul 13 19:58:06.659508 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:06.659475 2582 generic.go:356] "Generic (PLEG): container finished" podID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerID="08846042de83b35cd1fa125634ee8a128530eafd40cb15d4f0c696176438854f" exitCode=0 Jul 13 19:58:06.659646 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:06.659546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" event={"ID":"59b0e682-1252-4db1-813b-7458f0f82c3b","Type":"ContainerDied","Data":"08846042de83b35cd1fa125634ee8a128530eafd40cb15d4f0c696176438854f"} Jul 13 19:58:06.659646 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:06.659610 2582 scope.go:117] "RemoveContainer" containerID="a1276ee52c3bce7b99c8b0d40deac275d4094124441f9dfc902fe5e00650a6fb" Jul 13 19:58:07.692419 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:07.692388 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:58:07.804676 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:07.804651 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:58:07.832743 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:07.832718 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca\") pod \"59b0e682-1252-4db1-813b-7458f0f82c3b\" (UID: \"59b0e682-1252-4db1-813b-7458f0f82c3b\") " Jul 13 19:58:07.833045 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:07.833023 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59b0e682-1252-4db1-813b-7458f0f82c3b" (UID: "59b0e682-1252-4db1-813b-7458f0f82c3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:58:07.934236 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:07.934160 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b0e682-1252-4db1-813b-7458f0f82c3b-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:58:08.672572 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.672541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" event={"ID":"59b0e682-1252-4db1-813b-7458f0f82c3b","Type":"ContainerDied","Data":"88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd"} Jul 13 19:58:08.672763 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.672589 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b106782ec635f0ab1f2327999333a59757663c21e0394a52302573bcc96cfd" Jul 13 19:58:08.672763 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.672553 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p" Jul 13 19:58:08.704021 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.703988 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:58:08.708533 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.708511 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66d7f166-k9p5p"] Jul 13 19:58:08.961202 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:08.961076 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" path="/var/lib/kubelet/pods/59b0e682-1252-4db1-813b-7458f0f82c3b/volumes" Jul 13 19:58:57.893573 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:57.893541 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:58:57.893989 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:57.893743 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" podUID="9599026a-acfb-4367-865e-4061d9f4ad6e" containerName="maas-api" containerID="cri-o://b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf" gracePeriod=30 Jul 13 19:58:58.147387 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.147322 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:58:58.174552 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.174513 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca\") pod \"9599026a-acfb-4367-865e-4061d9f4ad6e\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " Jul 13 19:58:58.174704 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.174600 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jdbr\" (UniqueName: \"kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr\") pod \"9599026a-acfb-4367-865e-4061d9f4ad6e\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " Jul 13 19:58:58.174704 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.174678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls\") pod \"9599026a-acfb-4367-865e-4061d9f4ad6e\" (UID: \"9599026a-acfb-4367-865e-4061d9f4ad6e\") " Jul 13 19:58:58.174927 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.174904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9599026a-acfb-4367-865e-4061d9f4ad6e" (UID: "9599026a-acfb-4367-865e-4061d9f4ad6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 19:58:58.175048 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.175035 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9599026a-acfb-4367-865e-4061d9f4ad6e-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:58:58.176875 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.176846 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr" (OuterVolumeSpecName: "kube-api-access-9jdbr") pod "9599026a-acfb-4367-865e-4061d9f4ad6e" (UID: "9599026a-acfb-4367-865e-4061d9f4ad6e"). InnerVolumeSpecName "kube-api-access-9jdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 19:58:58.176875 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.176865 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9599026a-acfb-4367-865e-4061d9f4ad6e" (UID: "9599026a-acfb-4367-865e-4061d9f4ad6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 19:58:58.276014 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.275980 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdbr\" (UniqueName: \"kubernetes.io/projected/9599026a-acfb-4367-865e-4061d9f4ad6e-kube-api-access-9jdbr\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:58:58.276014 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.276009 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9599026a-acfb-4367-865e-4061d9f4ad6e-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 19:58:58.957829 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.957791 2582 generic.go:356] "Generic (PLEG): container finished" podID="9599026a-acfb-4367-865e-4061d9f4ad6e" containerID="b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf" exitCode=0 Jul 13 19:58:58.958330 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.958312 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" Jul 13 19:58:58.961761 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.961734 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" event={"ID":"9599026a-acfb-4367-865e-4061d9f4ad6e","Type":"ContainerDied","Data":"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf"} Jul 13 19:58:58.961886 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.961774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r" event={"ID":"9599026a-acfb-4367-865e-4061d9f4ad6e","Type":"ContainerDied","Data":"0d44d95ec5ea1f610db40bb483892c06ce560dfbaedb37c65c11d72d9d5b820b"} Jul 13 19:58:58.961886 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.961796 2582 scope.go:117] "RemoveContainer" containerID="b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf" Jul 13 19:58:58.971495 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.971480 2582 scope.go:117] "RemoveContainer" containerID="b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf" Jul 13 19:58:58.971733 ip-10-0-135-83 kubenswrapper[2582]: E0713 19:58:58.971713 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf\": container with ID starting with b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf not found: ID does not exist" containerID="b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf" Jul 13 19:58:58.971801 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.971745 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf"} err="failed to get container status \"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf\": rpc error: code = NotFound desc = could not find container \"b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf\": container with ID starting with b06f0c335e97d9c5303a449eb310e58bcd69b44088635f8f6bf55b23bf46b4bf not found: ID does not exist" Jul 13 19:58:58.985538 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.985507 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:58:58.990359 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:58:58.990335 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66d7f166-7fdfc7955b-n7p9r"] Jul 13 19:59:00.961085 ip-10-0-135-83 kubenswrapper[2582]: I0713 19:59:00.961052 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9599026a-acfb-4367-865e-4061d9f4ad6e" path="/var/lib/kubelet/pods/9599026a-acfb-4367-865e-4061d9f4ad6e/volumes" Jul 13 20:00:09.064466 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:09.064416 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:00:09.071128 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:09.071108 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:00:52.610468 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.610421 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:00:52.611038 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611018 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9599026a-acfb-4367-865e-4061d9f4ad6e" containerName="maas-api" Jul 13 20:00:52.611038 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611037 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9599026a-acfb-4367-865e-4061d9f4ad6e" containerName="maas-api" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611070 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611079 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611097 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611106 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611115 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611124 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611492 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611214 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611492 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611226 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611492 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611237 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b0e682-1252-4db1-813b-7458f0f82c3b" containerName="revoke-keys" Jul 13 20:00:52.611492 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.611248 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9599026a-acfb-4367-865e-4061d9f4ad6e" containerName="maas-api" Jul 13 20:00:52.614472 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.614452 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.617200 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.617174 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d94baa8e-serving-cert\"" Jul 13 20:00:52.617370 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.617231 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 20:00:52.617370 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.617243 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 20:00:52.618050 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.618034 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 20:00:52.627919 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.627887 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:00:52.689825 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.689792 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.689825 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.689831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwbs\" (UniqueName: \"kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.690051 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.689929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.790738 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.790703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.790738 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.790750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.790969 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.790768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwbs\" (UniqueName: \"kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.791398 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.791372 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.793327 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.793305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.799184 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.799160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwbs\" (UniqueName: \"kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs\") pod \"maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:52.929612 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:52.929534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:53.077342 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:53.077315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:00:53.079028 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:00:53.079000 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e9e7fb_2245_4a23_9753_14307e34c2c8.slice/crio-81b942edde3cb913c4c171801fc458a9ad172cf649407a8a01309a704d952286 WatchSource:0}: Error finding container 81b942edde3cb913c4c171801fc458a9ad172cf649407a8a01309a704d952286: Status 404 returned error can't find the container with id 81b942edde3cb913c4c171801fc458a9ad172cf649407a8a01309a704d952286 Jul 13 20:00:53.080325 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:53.080307 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 20:00:53.607496 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:53.607467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" event={"ID":"48e9e7fb-2245-4a23-9753-14307e34c2c8","Type":"ContainerStarted","Data":"81b942edde3cb913c4c171801fc458a9ad172cf649407a8a01309a704d952286"} Jul 13 20:00:54.614771 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:54.614739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" event={"ID":"48e9e7fb-2245-4a23-9753-14307e34c2c8","Type":"ContainerStarted","Data":"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09"} Jul 13 20:00:54.615261 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:54.614935 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:00:54.632405 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:00:54.632357 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" podStartSLOduration=2.151034994 podStartE2EDuration="2.632341772s" totalImagesPullingTime="481.306778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:00:54.631245482 +0000 UTC m=+2146.192643526" watchObservedRunningTime="2026-07-13 20:00:54.632341772 +0000 UTC m=+2146.193739816" Jul 13 20:01:00.626443 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:00.626390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:01:04.419893 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.419864 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf"] Jul 13 20:01:04.423535 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.423509 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:04.427709 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.427690 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:01:04.445381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.445353 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf"] Jul 13 20:01:04.500347 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.500317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf\" (UID: \"4170062d-38eb-4127-bcd9-10213996be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:04.601805 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.601771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf\" (UID: \"4170062d-38eb-4127-bcd9-10213996be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:04.602412 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.602391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf\" (UID: \"4170062d-38eb-4127-bcd9-10213996be97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:04.734088 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.734006 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:04.869924 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:04.869897 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf"] Jul 13 20:01:04.871783 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:01:04.871754 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4170062d_38eb_4127_bcd9_10213996be97.slice/crio-753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9 WatchSource:0}: Error finding container 753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9: Status 404 returned error can't find the container with id 753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9 Jul 13 20:01:05.678320 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:05.678290 2582 generic.go:356] "Generic (PLEG): container finished" podID="4170062d-38eb-4127-bcd9-10213996be97" containerID="1f7a92d29f103bd8de204749aff24752c1aaad5854473c0b7c29b417f7c18666" exitCode=0 Jul 13 20:01:05.678716 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:05.678355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" event={"ID":"4170062d-38eb-4127-bcd9-10213996be97","Type":"ContainerDied","Data":"1f7a92d29f103bd8de204749aff24752c1aaad5854473c0b7c29b417f7c18666"} Jul 13 20:01:05.678716 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:05.678397 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" event={"ID":"4170062d-38eb-4127-bcd9-10213996be97","Type":"ContainerStarted","Data":"753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9"} Jul 13 20:01:06.821170 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:06.821138 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:06.921887 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:06.921854 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca\") pod \"4170062d-38eb-4127-bcd9-10213996be97\" (UID: \"4170062d-38eb-4127-bcd9-10213996be97\") " Jul 13 20:01:06.922217 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:06.922192 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4170062d-38eb-4127-bcd9-10213996be97" (UID: "4170062d-38eb-4127-bcd9-10213996be97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:01:07.022759 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:07.022730 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4170062d-38eb-4127-bcd9-10213996be97-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:01:07.690144 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:07.690111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" event={"ID":"4170062d-38eb-4127-bcd9-10213996be97","Type":"ContainerDied","Data":"753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9"} Jul 13 20:01:07.690144 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:07.690144 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf" Jul 13 20:01:07.690370 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:07.690160 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753d9298501eeb4cdc3aa353db0893d58cf0dba2e4c69a19ec8abf25734956c9" Jul 13 20:01:14.767230 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:14.767195 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:01:14.767733 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:14.767465 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" podUID="48e9e7fb-2245-4a23-9753-14307e34c2c8" containerName="maas-api" containerID="cri-o://d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09" gracePeriod=30 Jul 13 20:01:15.023880 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.023820 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:01:15.194747 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.194716 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls\") pod \"48e9e7fb-2245-4a23-9753-14307e34c2c8\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " Jul 13 20:01:15.194921 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.194775 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca\") pod \"48e9e7fb-2245-4a23-9753-14307e34c2c8\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " Jul 13 20:01:15.194921 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.194832 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lwbs\" (UniqueName: \"kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs\") pod \"48e9e7fb-2245-4a23-9753-14307e34c2c8\" (UID: \"48e9e7fb-2245-4a23-9753-14307e34c2c8\") " Jul 13 20:01:15.195210 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.195187 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48e9e7fb-2245-4a23-9753-14307e34c2c8" (UID: "48e9e7fb-2245-4a23-9753-14307e34c2c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:01:15.196950 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.196923 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48e9e7fb-2245-4a23-9753-14307e34c2c8" (UID: "48e9e7fb-2245-4a23-9753-14307e34c2c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:01:15.197046 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.196980 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs" (OuterVolumeSpecName: "kube-api-access-9lwbs") pod "48e9e7fb-2245-4a23-9753-14307e34c2c8" (UID: "48e9e7fb-2245-4a23-9753-14307e34c2c8"). InnerVolumeSpecName "kube-api-access-9lwbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:01:15.295750 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.295675 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e9e7fb-2245-4a23-9753-14307e34c2c8-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:01:15.295750 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.295702 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwbs\" (UniqueName: \"kubernetes.io/projected/48e9e7fb-2245-4a23-9753-14307e34c2c8-kube-api-access-9lwbs\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:01:15.295750 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.295712 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e9e7fb-2245-4a23-9753-14307e34c2c8-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:01:15.738208 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.738172 2582 generic.go:356] "Generic (PLEG): container finished" podID="48e9e7fb-2245-4a23-9753-14307e34c2c8" containerID="d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09" exitCode=0 Jul 13 20:01:15.738381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.738227 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" Jul 13 20:01:15.738381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.738258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" event={"ID":"48e9e7fb-2245-4a23-9753-14307e34c2c8","Type":"ContainerDied","Data":"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09"} Jul 13 20:01:15.738381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.738343 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs" event={"ID":"48e9e7fb-2245-4a23-9753-14307e34c2c8","Type":"ContainerDied","Data":"81b942edde3cb913c4c171801fc458a9ad172cf649407a8a01309a704d952286"} Jul 13 20:01:15.738381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.738371 2582 scope.go:117] "RemoveContainer" containerID="d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09" Jul 13 20:01:15.748186 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.748166 2582 scope.go:117] "RemoveContainer" containerID="d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09" Jul 13 20:01:15.748441 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:01:15.748408 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09\": container with ID starting with d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09 not found: ID does not exist" containerID="d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09" Jul 13 20:01:15.748508 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.748489 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09"} err="failed to get container status \"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09\": rpc error: code = NotFound desc = could not find container \"d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09\": container with ID starting with d9d596fdd91db054cd1150a755c04f9b608faddc3a473f12cdfb4ddb71271a09 not found: ID does not exist" Jul 13 20:01:15.759568 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.759547 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:01:15.765105 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:15.765087 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d94baa8e-685f46b4db-nd6cs"] Jul 13 20:01:16.961446 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:16.961402 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e9e7fb-2245-4a23-9753-14307e34c2c8" path="/var/lib/kubelet/pods/48e9e7fb-2245-4a23-9753-14307e34c2c8/volumes" Jul 13 20:01:52.937434 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937391 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:01:52.937925 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937895 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4170062d-38eb-4127-bcd9-10213996be97" containerName="revoke-keys" Jul 13 20:01:52.937925 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937910 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4170062d-38eb-4127-bcd9-10213996be97" containerName="revoke-keys" Jul 13 20:01:52.937925 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937921 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e9e7fb-2245-4a23-9753-14307e34c2c8" containerName="maas-api" Jul 13 20:01:52.937925 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937926 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e9e7fb-2245-4a23-9753-14307e34c2c8" containerName="maas-api" Jul 13 20:01:52.938076 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937988 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e9e7fb-2245-4a23-9753-14307e34c2c8" containerName="maas-api" Jul 13 20:01:52.938076 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.937997 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4170062d-38eb-4127-bcd9-10213996be97" containerName="revoke-keys" Jul 13 20:01:52.941512 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.941492 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:52.943981 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.943957 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-02a26ce0-serving-cert\"" Jul 13 20:01:52.945162 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.945130 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 20:01:52.945309 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.945233 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 20:01:52.945309 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.945233 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 20:01:52.952783 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:52.952758 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:01:53.036257 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.036227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9glp\" (UniqueName: \"kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.036479 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.036291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.036557 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.036504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.137676 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.137639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.137853 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.137698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9glp\" (UniqueName: \"kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.137853 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.137723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.138286 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.138261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.140294 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.140274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.146093 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.146075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9glp\" (UniqueName: \"kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp\") pod \"maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.253149 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.253059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.393407 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.393379 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:01:53.394784 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:01:53.394761 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37ace84_dd22_4774_8c33_eb6d91c0a29f.slice/crio-4c679fe73eb7c9c8dec8d3eb4c813ce97db5878693bf2835b3dd500d72b92334 WatchSource:0}: Error finding container 4c679fe73eb7c9c8dec8d3eb4c813ce97db5878693bf2835b3dd500d72b92334: Status 404 returned error can't find the container with id 4c679fe73eb7c9c8dec8d3eb4c813ce97db5878693bf2835b3dd500d72b92334 Jul 13 20:01:53.968878 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.968837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" event={"ID":"e37ace84-dd22-4774-8c33-eb6d91c0a29f","Type":"ContainerStarted","Data":"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35"} Jul 13 20:01:53.969344 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.968897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" event={"ID":"e37ace84-dd22-4774-8c33-eb6d91c0a29f","Type":"ContainerStarted","Data":"4c679fe73eb7c9c8dec8d3eb4c813ce97db5878693bf2835b3dd500d72b92334"} Jul 13 20:01:53.969344 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.968924 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:01:53.987032 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:53.986980 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" podStartSLOduration=1.577929556 podStartE2EDuration="1.986966439s" totalImagesPullingTime="409.036883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:01:53.984445071 +0000 UTC m=+2205.545843106" watchObservedRunningTime="2026-07-13 20:01:53.986966439 +0000 UTC m=+2205.548364481" Jul 13 20:01:59.987511 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:01:59.987407 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:02:30.367979 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.367894 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc"] Jul 13 20:02:30.371419 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.371402 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:30.373826 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.373808 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:02:30.378762 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.378735 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc"] Jul 13 20:02:30.463045 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.463012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc\" (UID: \"8eb1e7f8-46b5-4832-a611-68e3fc7ec802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:30.563814 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.563784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc\" (UID: \"8eb1e7f8-46b5-4832-a611-68e3fc7ec802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:30.564394 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.564376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc\" (UID: \"8eb1e7f8-46b5-4832-a611-68e3fc7ec802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:30.683648 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.683558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:30.818390 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:02:30.818359 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb1e7f8_46b5_4832_a611_68e3fc7ec802.slice/crio-1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23 WatchSource:0}: Error finding container 1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23: Status 404 returned error can't find the container with id 1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23 Jul 13 20:02:30.818746 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:30.818721 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc"] Jul 13 20:02:31.186840 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:31.186809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" event={"ID":"8eb1e7f8-46b5-4832-a611-68e3fc7ec802","Type":"ContainerStarted","Data":"f38bf4ea313b9c537a5d70777aa3e731c87e80226c60a2e23108e8aa0dda3e95"} Jul 13 20:02:31.187010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:31.186860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" event={"ID":"8eb1e7f8-46b5-4832-a611-68e3fc7ec802","Type":"ContainerStarted","Data":"1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23"} Jul 13 20:02:31.203476 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:31.203363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" podStartSLOduration=1.203347961 podStartE2EDuration="1.203347961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:02:31.200970842 +0000 UTC m=+2242.762368899" watchObservedRunningTime="2026-07-13 20:02:31.203347961 +0000 UTC m=+2242.764746005" Jul 13 20:02:32.193852 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:32.193816 2582 generic.go:356] "Generic (PLEG): container finished" podID="8eb1e7f8-46b5-4832-a611-68e3fc7ec802" containerID="f38bf4ea313b9c537a5d70777aa3e731c87e80226c60a2e23108e8aa0dda3e95" exitCode=0 Jul 13 20:02:32.194226 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:32.193901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" event={"ID":"8eb1e7f8-46b5-4832-a611-68e3fc7ec802","Type":"ContainerDied","Data":"f38bf4ea313b9c537a5d70777aa3e731c87e80226c60a2e23108e8aa0dda3e95"} Jul 13 20:02:33.333246 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:33.333222 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:33.389092 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:33.389053 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca\") pod \"8eb1e7f8-46b5-4832-a611-68e3fc7ec802\" (UID: \"8eb1e7f8-46b5-4832-a611-68e3fc7ec802\") " Jul 13 20:02:33.389379 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:33.389357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8eb1e7f8-46b5-4832-a611-68e3fc7ec802" (UID: "8eb1e7f8-46b5-4832-a611-68e3fc7ec802"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:02:33.490404 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:33.490322 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb1e7f8-46b5-4832-a611-68e3fc7ec802-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:02:34.206181 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:34.206151 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" Jul 13 20:02:34.206355 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:34.206186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc" event={"ID":"8eb1e7f8-46b5-4832-a611-68e3fc7ec802","Type":"ContainerDied","Data":"1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23"} Jul 13 20:02:34.206355 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:34.206226 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b547ff0fa58edae7d02f32804c088df568e5ef32ba6922979da7117545d3c23" Jul 13 20:02:41.246153 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.246118 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:02:41.246627 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.246369 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" podUID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" containerName="maas-api" containerID="cri-o://3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35" gracePeriod=30 Jul 13 20:02:41.501287 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.501227 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:02:41.559819 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.559787 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca\") pod \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " Jul 13 20:02:41.559975 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.559843 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9glp\" (UniqueName: \"kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp\") pod \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " Jul 13 20:02:41.559975 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.559879 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls\") pod \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\" (UID: \"e37ace84-dd22-4774-8c33-eb6d91c0a29f\") " Jul 13 20:02:41.560157 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.560128 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e37ace84-dd22-4774-8c33-eb6d91c0a29f" (UID: "e37ace84-dd22-4774-8c33-eb6d91c0a29f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:02:41.562078 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.562055 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp" (OuterVolumeSpecName: "kube-api-access-t9glp") pod "e37ace84-dd22-4774-8c33-eb6d91c0a29f" (UID: "e37ace84-dd22-4774-8c33-eb6d91c0a29f"). InnerVolumeSpecName "kube-api-access-t9glp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:02:41.562244 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.562226 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e37ace84-dd22-4774-8c33-eb6d91c0a29f" (UID: "e37ace84-dd22-4774-8c33-eb6d91c0a29f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:02:41.660905 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.660868 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37ace84-dd22-4774-8c33-eb6d91c0a29f-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:02:41.660905 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.660898 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9glp\" (UniqueName: \"kubernetes.io/projected/e37ace84-dd22-4774-8c33-eb6d91c0a29f-kube-api-access-t9glp\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:02:41.660905 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:41.660908 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37ace84-dd22-4774-8c33-eb6d91c0a29f-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:02:42.256235 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.256206 2582 generic.go:356] "Generic (PLEG): container finished" podID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" containerID="3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35" exitCode=0 Jul 13 20:02:42.256656 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.256255 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" event={"ID":"e37ace84-dd22-4774-8c33-eb6d91c0a29f","Type":"ContainerDied","Data":"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35"} Jul 13 20:02:42.256656 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.256293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" event={"ID":"e37ace84-dd22-4774-8c33-eb6d91c0a29f","Type":"ContainerDied","Data":"4c679fe73eb7c9c8dec8d3eb4c813ce97db5878693bf2835b3dd500d72b92334"} Jul 13 20:02:42.256656 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.256314 2582 scope.go:117] "RemoveContainer" containerID="3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35" Jul 13 20:02:42.256656 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.256256 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5" Jul 13 20:02:42.266152 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.266134 2582 scope.go:117] "RemoveContainer" containerID="3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35" Jul 13 20:02:42.266466 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:02:42.266416 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35\": container with ID starting with 3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35 not found: ID does not exist" containerID="3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35" Jul 13 20:02:42.266530 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.266464 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35"} err="failed to get container status \"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35\": rpc error: code = NotFound desc = could not find container \"3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35\": container with ID starting with 3910eb8cf9cfe5573a13f83fdbf794b9385cecbe4e8cc2c4e2b27b8aec91cb35 not found: ID does not exist" Jul 13 20:02:42.278287 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.278264 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:02:42.282063 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.282043 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-02a26ce0-cc4c5cc5c-257d5"] Jul 13 20:02:42.960707 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:02:42.960672 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" path="/var/lib/kubelet/pods/e37ace84-dd22-4774-8c33-eb6d91c0a29f/volumes" Jul 13 20:04:09.170667 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:09.170637 2582 scope.go:117] "RemoveContainer" containerID="08846042de83b35cd1fa125634ee8a128530eafd40cb15d4f0c696176438854f" Jul 13 20:04:35.808566 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.808526 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809077 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb1e7f8-46b5-4832-a611-68e3fc7ec802" containerName="revoke-keys" Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809092 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb1e7f8-46b5-4832-a611-68e3fc7ec802" containerName="revoke-keys" Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809122 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" containerName="maas-api" Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809128 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" containerName="maas-api" Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809215 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb1e7f8-46b5-4832-a611-68e3fc7ec802" containerName="revoke-keys" Jul 13 20:04:35.811053 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.809224 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e37ace84-dd22-4774-8c33-eb6d91c0a29f" containerName="maas-api" Jul 13 20:04:35.812640 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.812623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.815191 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.815166 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-416f95-serving-cert\"" Jul 13 20:04:35.815464 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.815257 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 20:04:35.815591 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.815297 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 20:04:35.815659 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.815297 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 20:04:35.822852 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.822828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:04:35.874808 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.874778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.875007 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.874835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.875007 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.874906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdxb\" (UniqueName: \"kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.976069 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.976030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.976241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.976075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.976241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.976195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdxb\" (UniqueName: \"kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.976845 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.976823 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.978803 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.978778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:35.985179 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:35.985156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdxb\" (UniqueName: \"kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb\") pod \"maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:36.125592 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.125552 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:36.266648 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.266623 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:04:36.267574 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:04:36.267549 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274616b5_fa94_458e_bac3_30e84e898772.slice/crio-bbb9ef910a15630cc0707c5a13fc24e5e7b09727b14c5c2aa4f9f032994a0c99 WatchSource:0}: Error finding container bbb9ef910a15630cc0707c5a13fc24e5e7b09727b14c5c2aa4f9f032994a0c99: Status 404 returned error can't find the container with id bbb9ef910a15630cc0707c5a13fc24e5e7b09727b14c5c2aa4f9f032994a0c99 Jul 13 20:04:36.888359 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.888322 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" event={"ID":"274616b5-fa94-458e-bac3-30e84e898772","Type":"ContainerStarted","Data":"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b"} Jul 13 20:04:36.888701 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.888390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" event={"ID":"274616b5-fa94-458e-bac3-30e84e898772","Type":"ContainerStarted","Data":"bbb9ef910a15630cc0707c5a13fc24e5e7b09727b14c5c2aa4f9f032994a0c99"} Jul 13 20:04:36.888701 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.888436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:36.905689 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:36.905648 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" podStartSLOduration=1.348242114 podStartE2EDuration="1.905633112s" totalImagesPullingTime="557.390998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:04:36.904123284 +0000 UTC m=+2368.465521340" watchObservedRunningTime="2026-07-13 20:04:36.905633112 +0000 UTC m=+2368.467031156" Jul 13 20:04:42.900096 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:42.900060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:04:43.604993 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.604961 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:04:43.608837 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.608811 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.611320 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.611296 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6d9be2-serving-cert\"" Jul 13 20:04:43.621350 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.621321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:04:43.749211 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.749171 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.749410 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.749289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.749410 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.749330 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrmf\" (UniqueName: \"kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.850862 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.850820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.851036 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.850906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.851036 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.850929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrmf\" (UniqueName: \"kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.851509 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.851488 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.853642 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.853616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.859147 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.859092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrmf\" (UniqueName: \"kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf\") pod \"maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:43.920947 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:43.920915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:44.066571 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:44.066545 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:04:44.067846 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:04:44.067818 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbdd3a9_6907_4aa2_9c72_068d37871183.slice/crio-78c544c5a67b0827f8efa053f6c9b3904c0ae073770c23febdcee5fa11d09517 WatchSource:0}: Error finding container 78c544c5a67b0827f8efa053f6c9b3904c0ae073770c23febdcee5fa11d09517: Status 404 returned error can't find the container with id 78c544c5a67b0827f8efa053f6c9b3904c0ae073770c23febdcee5fa11d09517 Jul 13 20:04:44.937885 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:44.937845 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" event={"ID":"6fbdd3a9-6907-4aa2-9c72-068d37871183","Type":"ContainerStarted","Data":"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb"} Jul 13 20:04:44.938255 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:44.937908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" event={"ID":"6fbdd3a9-6907-4aa2-9c72-068d37871183","Type":"ContainerStarted","Data":"78c544c5a67b0827f8efa053f6c9b3904c0ae073770c23febdcee5fa11d09517"} Jul 13 20:04:44.938255 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:44.937988 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:04:44.957236 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:44.957187 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" podStartSLOduration=1.431786565 podStartE2EDuration="1.957168471s" totalImagesPullingTime="525.381906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:04:44.954338481 +0000 UTC m=+2376.515736524" watchObservedRunningTime="2026-07-13 20:04:44.957168471 +0000 UTC m=+2376.518566515" Jul 13 20:04:50.950256 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:04:50.950229 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:05:09.102163 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:09.102137 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:05:09.108581 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:09.108558 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:05:47.706154 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.706072 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk"] Jul 13 20:05:47.709732 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.709714 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:47.712061 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.712042 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:05:47.717358 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.717023 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk"] Jul 13 20:05:47.727866 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.727839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk\" (UID: \"9b7dd2c1-54ce-4351-ac62-5e41b52b4676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:47.828668 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.828634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk\" (UID: \"9b7dd2c1-54ce-4351-ac62-5e41b52b4676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:47.829232 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:47.829212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk\" (UID: \"9b7dd2c1-54ce-4351-ac62-5e41b52b4676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:48.021811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:48.021723 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:48.169150 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:48.169125 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk"] Jul 13 20:05:48.171771 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:05:48.171748 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7dd2c1_54ce_4351_ac62_5e41b52b4676.slice/crio-de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44 WatchSource:0}: Error finding container de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44: Status 404 returned error can't find the container with id de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44 Jul 13 20:05:48.305658 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:48.305571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" event={"ID":"9b7dd2c1-54ce-4351-ac62-5e41b52b4676","Type":"ContainerStarted","Data":"de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44"} Jul 13 20:05:49.312752 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:49.312718 2582 generic.go:356] "Generic (PLEG): container finished" podID="9b7dd2c1-54ce-4351-ac62-5e41b52b4676" containerID="ad52e8b06dc0144d58411e80b62537dbd728b1e24b654606f14f3dff9b8852ba" exitCode=0 Jul 13 20:05:49.313192 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:49.312806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" event={"ID":"9b7dd2c1-54ce-4351-ac62-5e41b52b4676","Type":"ContainerDied","Data":"ad52e8b06dc0144d58411e80b62537dbd728b1e24b654606f14f3dff9b8852ba"} Jul 13 20:05:50.456291 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:50.456261 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:50.551739 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:50.551704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca\") pod \"9b7dd2c1-54ce-4351-ac62-5e41b52b4676\" (UID: \"9b7dd2c1-54ce-4351-ac62-5e41b52b4676\") " Jul 13 20:05:50.552078 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:50.552054 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b7dd2c1-54ce-4351-ac62-5e41b52b4676" (UID: "9b7dd2c1-54ce-4351-ac62-5e41b52b4676"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:05:50.652612 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:50.652576 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7dd2c1-54ce-4351-ac62-5e41b52b4676-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:05:51.324038 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:51.323951 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" Jul 13 20:05:51.324038 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:51.323998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk" event={"ID":"9b7dd2c1-54ce-4351-ac62-5e41b52b4676","Type":"ContainerDied","Data":"de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44"} Jul 13 20:05:51.324269 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:51.324043 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1d948d213c8d9fd0ae295ea3bb6e424fde5ce2fa61e21b8ee282c8dfdd8e44" Jul 13 20:05:58.059285 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.059247 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:05:58.059951 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.059532 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" podUID="274616b5-fa94-458e-bac3-30e84e898772" containerName="maas-api" containerID="cri-o://43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b" gracePeriod=30 Jul 13 20:05:58.307597 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.307574 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:05:58.363659 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.363627 2582 generic.go:356] "Generic (PLEG): container finished" podID="274616b5-fa94-458e-bac3-30e84e898772" containerID="43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b" exitCode=0 Jul 13 20:05:58.363811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.363672 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" Jul 13 20:05:58.363811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.363716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" event={"ID":"274616b5-fa94-458e-bac3-30e84e898772","Type":"ContainerDied","Data":"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b"} Jul 13 20:05:58.363811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.363772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt" event={"ID":"274616b5-fa94-458e-bac3-30e84e898772","Type":"ContainerDied","Data":"bbb9ef910a15630cc0707c5a13fc24e5e7b09727b14c5c2aa4f9f032994a0c99"} Jul 13 20:05:58.363811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.363796 2582 scope.go:117] "RemoveContainer" containerID="43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b" Jul 13 20:05:58.372978 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.372957 2582 scope.go:117] "RemoveContainer" containerID="43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b" Jul 13 20:05:58.373222 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:05:58.373206 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b\": container with ID starting with 43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b not found: ID does not exist" containerID="43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b" Jul 13 20:05:58.373270 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.373230 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b"} err="failed to get container status \"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b\": rpc error: code = NotFound desc = could not find container \"43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b\": container with ID starting with 43907b55b211fc8913b195f45a2644a95867de3e6788b08b98c32355b4a04c8b not found: ID does not exist" Jul 13 20:05:58.417828 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.417800 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca\") pod \"274616b5-fa94-458e-bac3-30e84e898772\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " Jul 13 20:05:58.417941 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.417834 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kdxb\" (UniqueName: \"kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb\") pod \"274616b5-fa94-458e-bac3-30e84e898772\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " Jul 13 20:05:58.417941 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.417871 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls\") pod \"274616b5-fa94-458e-bac3-30e84e898772\" (UID: \"274616b5-fa94-458e-bac3-30e84e898772\") " Jul 13 20:05:58.418219 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.418192 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "274616b5-fa94-458e-bac3-30e84e898772" (UID: "274616b5-fa94-458e-bac3-30e84e898772"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:05:58.420005 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.419985 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "274616b5-fa94-458e-bac3-30e84e898772" (UID: "274616b5-fa94-458e-bac3-30e84e898772"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:05:58.420123 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.420108 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb" (OuterVolumeSpecName: "kube-api-access-9kdxb") pod "274616b5-fa94-458e-bac3-30e84e898772" (UID: "274616b5-fa94-458e-bac3-30e84e898772"). InnerVolumeSpecName "kube-api-access-9kdxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:05:58.519003 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.518966 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274616b5-fa94-458e-bac3-30e84e898772-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:05:58.519003 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.518993 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274616b5-fa94-458e-bac3-30e84e898772-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:05:58.519003 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.519004 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdxb\" (UniqueName: \"kubernetes.io/projected/274616b5-fa94-458e-bac3-30e84e898772-kube-api-access-9kdxb\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:05:58.686393 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.686361 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:05:58.689778 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.689759 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-416f95-7cd67bdb4c-pd6vt"] Jul 13 20:05:58.962655 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:05:58.962577 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274616b5-fa94-458e-bac3-30e84e898772" path="/var/lib/kubelet/pods/274616b5-fa94-458e-bac3-30e84e898772/volumes" Jul 13 20:06:13.819140 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819106 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t"] Jul 13 20:06:13.819546 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819518 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274616b5-fa94-458e-bac3-30e84e898772" containerName="maas-api" Jul 13 20:06:13.819546 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819530 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="274616b5-fa94-458e-bac3-30e84e898772" containerName="maas-api" Jul 13 20:06:13.819621 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819549 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7dd2c1-54ce-4351-ac62-5e41b52b4676" containerName="revoke-keys" Jul 13 20:06:13.819621 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819554 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7dd2c1-54ce-4351-ac62-5e41b52b4676" containerName="revoke-keys" Jul 13 20:06:13.819621 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819614 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7dd2c1-54ce-4351-ac62-5e41b52b4676" containerName="revoke-keys" Jul 13 20:06:13.819714 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.819624 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="274616b5-fa94-458e-bac3-30e84e898772" containerName="maas-api" Jul 13 20:06:13.824344 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.824327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:13.827241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.827215 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:06:13.830414 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.830383 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t"] Jul 13 20:06:13.958043 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:13.958007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t\" (UID: \"ad237892-6781-4d28-ad9b-6cddda7f2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:14.059194 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:14.059161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t\" (UID: \"ad237892-6781-4d28-ad9b-6cddda7f2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:14.059828 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:14.059808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t\" (UID: \"ad237892-6781-4d28-ad9b-6cddda7f2bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:14.137471 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:14.137411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:14.276143 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:14.276113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t"] Jul 13 20:06:14.277838 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:06:14.277812 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad237892_6781_4d28_ad9b_6cddda7f2bed.slice/crio-43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b WatchSource:0}: Error finding container 43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b: Status 404 returned error can't find the container with id 43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b Jul 13 20:06:14.459940 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:14.459902 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" event={"ID":"ad237892-6781-4d28-ad9b-6cddda7f2bed","Type":"ContainerStarted","Data":"43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b"} Jul 13 20:06:15.466657 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:15.466623 2582 generic.go:356] "Generic (PLEG): container finished" podID="ad237892-6781-4d28-ad9b-6cddda7f2bed" containerID="b8969efdf19a6e5914159df60a36415af7f039348b7857d911fd7ef1d7266dc0" exitCode=0 Jul 13 20:06:15.467057 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:15.466711 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" event={"ID":"ad237892-6781-4d28-ad9b-6cddda7f2bed","Type":"ContainerDied","Data":"b8969efdf19a6e5914159df60a36415af7f039348b7857d911fd7ef1d7266dc0"} Jul 13 20:06:16.609386 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:16.609365 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:16.684268 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:16.684230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca\") pod \"ad237892-6781-4d28-ad9b-6cddda7f2bed\" (UID: \"ad237892-6781-4d28-ad9b-6cddda7f2bed\") " Jul 13 20:06:16.684622 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:16.684599 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad237892-6781-4d28-ad9b-6cddda7f2bed" (UID: "ad237892-6781-4d28-ad9b-6cddda7f2bed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:06:16.785457 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:16.785349 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad237892-6781-4d28-ad9b-6cddda7f2bed-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:06:17.479887 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:17.479855 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" Jul 13 20:06:17.480035 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:17.479898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t" event={"ID":"ad237892-6781-4d28-ad9b-6cddda7f2bed","Type":"ContainerDied","Data":"43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b"} Jul 13 20:06:17.480035 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:17.479968 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43b3c9bd476c7464d61375ef43042f1f9dc5c8f5ae3aefa32a1769df2ab9c12b" Jul 13 20:06:24.226776 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.226738 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:06:24.227321 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.227004 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" podUID="6fbdd3a9-6907-4aa2-9c72-068d37871183" containerName="maas-api" containerID="cri-o://6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb" gracePeriod=30 Jul 13 20:06:24.476660 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.476634 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:06:24.520626 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.520530 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbdd3a9-6907-4aa2-9c72-068d37871183" containerID="6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb" exitCode=0 Jul 13 20:06:24.520626 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.520569 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" Jul 13 20:06:24.520626 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.520617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" event={"ID":"6fbdd3a9-6907-4aa2-9c72-068d37871183","Type":"ContainerDied","Data":"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb"} Jul 13 20:06:24.520871 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.520666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9" event={"ID":"6fbdd3a9-6907-4aa2-9c72-068d37871183","Type":"ContainerDied","Data":"78c544c5a67b0827f8efa053f6c9b3904c0ae073770c23febdcee5fa11d09517"} Jul 13 20:06:24.520871 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.520695 2582 scope.go:117] "RemoveContainer" containerID="6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb" Jul 13 20:06:24.530055 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.530035 2582 scope.go:117] "RemoveContainer" containerID="6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb" Jul 13 20:06:24.530332 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:06:24.530313 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb\": container with ID starting with 6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb not found: ID does not exist" containerID="6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb" Jul 13 20:06:24.530391 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.530346 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb"} err="failed to get container status \"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb\": rpc error: code = NotFound desc = could not find container \"6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb\": container with ID starting with 6053eca3fc90ae267ad7e80c88e8870515376af5c603db77b2d044c0b44fbebb not found: ID does not exist" Jul 13 20:06:24.552759 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.552727 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcrmf\" (UniqueName: \"kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf\") pod \"6fbdd3a9-6907-4aa2-9c72-068d37871183\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " Jul 13 20:06:24.552910 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.552769 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls\") pod \"6fbdd3a9-6907-4aa2-9c72-068d37871183\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " Jul 13 20:06:24.552910 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.552802 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca\") pod \"6fbdd3a9-6907-4aa2-9c72-068d37871183\" (UID: \"6fbdd3a9-6907-4aa2-9c72-068d37871183\") " Jul 13 20:06:24.553178 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.553157 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6fbdd3a9-6907-4aa2-9c72-068d37871183" (UID: "6fbdd3a9-6907-4aa2-9c72-068d37871183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:06:24.555094 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.555065 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf" (OuterVolumeSpecName: "kube-api-access-hcrmf") pod "6fbdd3a9-6907-4aa2-9c72-068d37871183" (UID: "6fbdd3a9-6907-4aa2-9c72-068d37871183"). InnerVolumeSpecName "kube-api-access-hcrmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:06:24.555189 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.555152 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6fbdd3a9-6907-4aa2-9c72-068d37871183" (UID: "6fbdd3a9-6907-4aa2-9c72-068d37871183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:06:24.654501 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.654462 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrmf\" (UniqueName: \"kubernetes.io/projected/6fbdd3a9-6907-4aa2-9c72-068d37871183-kube-api-access-hcrmf\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:06:24.654501 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.654494 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbdd3a9-6907-4aa2-9c72-068d37871183-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:06:24.654501 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.654504 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbdd3a9-6907-4aa2-9c72-068d37871183-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:06:24.846063 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.846016 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:06:24.848945 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.848915 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9be2-5d4bfc6b47-9rhh9"] Jul 13 20:06:24.961629 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:24.961598 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbdd3a9-6907-4aa2-9c72-068d37871183" path="/var/lib/kubelet/pods/6fbdd3a9-6907-4aa2-9c72-068d37871183/volumes" Jul 13 20:06:43.411764 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.408350 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:06:43.412383 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412258 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbdd3a9-6907-4aa2-9c72-068d37871183" containerName="maas-api" Jul 13 20:06:43.412383 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412287 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbdd3a9-6907-4aa2-9c72-068d37871183" containerName="maas-api" Jul 13 20:06:43.412383 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412316 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad237892-6781-4d28-ad9b-6cddda7f2bed" containerName="revoke-keys" Jul 13 20:06:43.412383 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412322 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad237892-6781-4d28-ad9b-6cddda7f2bed" containerName="revoke-keys" Jul 13 20:06:43.412383 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412384 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad237892-6781-4d28-ad9b-6cddda7f2bed" containerName="revoke-keys" Jul 13 20:06:43.412636 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.412395 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbdd3a9-6907-4aa2-9c72-068d37871183" containerName="maas-api" Jul 13 20:06:43.417166 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.417138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.419768 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.419694 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ebcb0c-serving-cert\"" Jul 13 20:06:43.419768 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.419726 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 20:06:43.419979 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.419726 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 20:06:43.420650 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.420628 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 20:06:43.421554 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.421534 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:06:43.423173 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.423148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdv5\" (UniqueName: \"kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.423273 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.423226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.423334 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.423316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.524234 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.524195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.524234 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.524236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdv5\" (UniqueName: \"kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.524510 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.524278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.524953 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.524928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.526825 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.526796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.532167 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.532144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdv5\" (UniqueName: \"kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5\") pod \"maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.730034 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.729944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:43.885780 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.885733 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:06:43.887343 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:06:43.887294 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b8e1de_9ae4_4493_b170_a96282d6cd7a.slice/crio-b4fe12bf140af3b611a7eae943e98a408bffef7a347668a16bb9150ed773c245 WatchSource:0}: Error finding container b4fe12bf140af3b611a7eae943e98a408bffef7a347668a16bb9150ed773c245: Status 404 returned error can't find the container with id b4fe12bf140af3b611a7eae943e98a408bffef7a347668a16bb9150ed773c245 Jul 13 20:06:43.888955 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:43.888938 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 20:06:44.645499 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:44.645410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" event={"ID":"96b8e1de-9ae4-4493-b170-a96282d6cd7a","Type":"ContainerStarted","Data":"b4fe12bf140af3b611a7eae943e98a408bffef7a347668a16bb9150ed773c245"} Jul 13 20:06:45.652962 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:45.652924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" event={"ID":"96b8e1de-9ae4-4493-b170-a96282d6cd7a","Type":"ContainerStarted","Data":"fab57503389efd7b3d313031292819a39c6cf4adc99c75d32bcba171e67cb80b"} Jul 13 20:06:45.653468 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:45.653055 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:45.672712 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:45.672653 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" podStartSLOduration=1.936654802 podStartE2EDuration="2.672636324s" totalImagesPullingTime="735.981522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:06:45.670059844 +0000 UTC m=+2497.231457911" watchObservedRunningTime="2026-07-13 20:06:45.672636324 +0000 UTC m=+2497.234034367" Jul 13 20:06:51.350909 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.350871 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:06:51.354711 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.354691 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.357117 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.357085 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3e52c5-serving-cert\"" Jul 13 20:06:51.369928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.369900 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:06:51.391385 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.391361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.391544 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.391467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.391544 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.391518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7lb\" (UniqueName: \"kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.492663 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.492630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.492848 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.492697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7lb\" (UniqueName: \"kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.492848 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.492716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.493373 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.493348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.495221 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.495199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.500720 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.500700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7lb\" (UniqueName: \"kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb\") pod \"maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.664789 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.664700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:06:51.665029 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.665010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:51.812039 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:51.812008 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:06:51.812863 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:06:51.812834 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c97f6f_c95b_4abd_893c_83a1c6880120.slice/crio-336285d8fce504f9b9a99aba11bdabe392ded41b82fd407f27061fd3f34f9a48 WatchSource:0}: Error finding container 336285d8fce504f9b9a99aba11bdabe392ded41b82fd407f27061fd3f34f9a48: Status 404 returned error can't find the container with id 336285d8fce504f9b9a99aba11bdabe392ded41b82fd407f27061fd3f34f9a48 Jul 13 20:06:52.696669 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:52.696574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" event={"ID":"c4c97f6f-c95b-4abd-893c-83a1c6880120","Type":"ContainerStarted","Data":"d066061cf0fde0c444dd608609e87f675b70d56498d80697d7c1cfeab82202aa"} Jul 13 20:06:52.696669 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:52.696631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" event={"ID":"c4c97f6f-c95b-4abd-893c-83a1c6880120","Type":"ContainerStarted","Data":"336285d8fce504f9b9a99aba11bdabe392ded41b82fd407f27061fd3f34f9a48"} Jul 13 20:06:52.697158 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:52.696712 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:06:52.717024 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:52.716962 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" podStartSLOduration=1.096887552 podStartE2EDuration="1.716947722s" totalImagesPullingTime="620.06017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:06:52.715038554 +0000 UTC m=+2504.276436602" watchObservedRunningTime="2026-07-13 20:06:52.716947722 +0000 UTC m=+2504.278345765" Jul 13 20:06:58.707979 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:06:58.707948 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:07:03.846655 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:03.846622 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn"] Jul 13 20:07:03.850389 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:03.850371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:03.853598 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:03.853561 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:07:03.862347 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:03.862320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn"] Jul 13 20:07:03.902834 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:03.902801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn\" (UID: \"e59db086-2f35-4625-b6c0-69613a619f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:04.004091 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.004062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn\" (UID: \"e59db086-2f35-4625-b6c0-69613a619f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:04.004739 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.004719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn\" (UID: \"e59db086-2f35-4625-b6c0-69613a619f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:04.162581 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.162552 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:04.302613 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.302581 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn"] Jul 13 20:07:04.303005 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:07:04.302983 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59db086_2f35_4625_b6c0_69613a619f83.slice/crio-a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4 WatchSource:0}: Error finding container a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4: Status 404 returned error can't find the container with id a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4 Jul 13 20:07:04.766661 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.766627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" event={"ID":"e59db086-2f35-4625-b6c0-69613a619f83","Type":"ContainerStarted","Data":"9018ff1096fbe986b0545fe618549711a99f3c51f11691a824021adfa129b9ea"} Jul 13 20:07:04.766842 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.766682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" event={"ID":"e59db086-2f35-4625-b6c0-69613a619f83","Type":"ContainerStarted","Data":"a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4"} Jul 13 20:07:04.784079 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:04.784028 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" podStartSLOduration=1.784013094 podStartE2EDuration="1.784013094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:07:04.781940416 +0000 UTC m=+2516.343338484" watchObservedRunningTime="2026-07-13 20:07:04.784013094 +0000 UTC m=+2516.345411137" Jul 13 20:07:05.773943 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:05.773908 2582 generic.go:356] "Generic (PLEG): container finished" podID="e59db086-2f35-4625-b6c0-69613a619f83" containerID="9018ff1096fbe986b0545fe618549711a99f3c51f11691a824021adfa129b9ea" exitCode=0 Jul 13 20:07:05.773943 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:05.773947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" event={"ID":"e59db086-2f35-4625-b6c0-69613a619f83","Type":"ContainerDied","Data":"9018ff1096fbe986b0545fe618549711a99f3c51f11691a824021adfa129b9ea"} Jul 13 20:07:06.915329 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:06.915308 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:07.032299 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.032219 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca\") pod \"e59db086-2f35-4625-b6c0-69613a619f83\" (UID: \"e59db086-2f35-4625-b6c0-69613a619f83\") " Jul 13 20:07:07.032567 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.032545 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e59db086-2f35-4625-b6c0-69613a619f83" (UID: "e59db086-2f35-4625-b6c0-69613a619f83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:07:07.133160 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.133134 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59db086-2f35-4625-b6c0-69613a619f83-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:07.786388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.786361 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" Jul 13 20:07:07.786568 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.786384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn" event={"ID":"e59db086-2f35-4625-b6c0-69613a619f83","Type":"ContainerDied","Data":"a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4"} Jul 13 20:07:07.786568 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:07.786478 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86e27d70ff1a98355bba30ac4c7e2f25715f841bc040e5b025220cb045a71e4" Jul 13 20:07:14.595529 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:14.595494 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:07:14.596058 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:14.595749 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" podUID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" containerName="maas-api" containerID="cri-o://fab57503389efd7b3d313031292819a39c6cf4adc99c75d32bcba171e67cb80b" gracePeriod=30 Jul 13 20:07:14.852106 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:14.852077 2582 generic.go:356] "Generic (PLEG): container finished" podID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" containerID="fab57503389efd7b3d313031292819a39c6cf4adc99c75d32bcba171e67cb80b" exitCode=0 Jul 13 20:07:14.852231 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:14.852149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" event={"ID":"96b8e1de-9ae4-4493-b170-a96282d6cd7a","Type":"ContainerDied","Data":"fab57503389efd7b3d313031292819a39c6cf4adc99c75d32bcba171e67cb80b"} Jul 13 20:07:14.871252 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:14.871231 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:07:15.000720 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.000688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls\") pod \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " Jul 13 20:07:15.000900 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.000797 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmdv5\" (UniqueName: \"kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5\") pod \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " Jul 13 20:07:15.000900 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.000825 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca\") pod \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\" (UID: \"96b8e1de-9ae4-4493-b170-a96282d6cd7a\") " Jul 13 20:07:15.001270 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.001239 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "96b8e1de-9ae4-4493-b170-a96282d6cd7a" (UID: "96b8e1de-9ae4-4493-b170-a96282d6cd7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:07:15.002979 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.002953 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5" (OuterVolumeSpecName: "kube-api-access-pmdv5") pod "96b8e1de-9ae4-4493-b170-a96282d6cd7a" (UID: "96b8e1de-9ae4-4493-b170-a96282d6cd7a"). InnerVolumeSpecName "kube-api-access-pmdv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:07:15.003092 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.003068 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "96b8e1de-9ae4-4493-b170-a96282d6cd7a" (UID: "96b8e1de-9ae4-4493-b170-a96282d6cd7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:07:15.101753 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.101675 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b8e1de-9ae4-4493-b170-a96282d6cd7a-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:15.101753 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.101710 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96b8e1de-9ae4-4493-b170-a96282d6cd7a-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:15.101753 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.101721 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdv5\" (UniqueName: \"kubernetes.io/projected/96b8e1de-9ae4-4493-b170-a96282d6cd7a-kube-api-access-pmdv5\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:15.858881 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.858851 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" Jul 13 20:07:15.859302 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.858885 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr" event={"ID":"96b8e1de-9ae4-4493-b170-a96282d6cd7a","Type":"ContainerDied","Data":"b4fe12bf140af3b611a7eae943e98a408bffef7a347668a16bb9150ed773c245"} Jul 13 20:07:15.859302 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.858942 2582 scope.go:117] "RemoveContainer" containerID="fab57503389efd7b3d313031292819a39c6cf4adc99c75d32bcba171e67cb80b" Jul 13 20:07:15.885118 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.885090 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:07:15.889765 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:15.889743 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ebcb0c-85b9bb8d96-xvqzr"] Jul 13 20:07:16.961177 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:16.961144 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" path="/var/lib/kubelet/pods/96b8e1de-9ae4-4493-b170-a96282d6cd7a/volumes" Jul 13 20:07:29.956463 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.956411 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv"] Jul 13 20:07:29.957049 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957028 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59db086-2f35-4625-b6c0-69613a619f83" containerName="revoke-keys" Jul 13 20:07:29.957136 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957051 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59db086-2f35-4625-b6c0-69613a619f83" containerName="revoke-keys" Jul 13 20:07:29.957136 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957072 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" containerName="maas-api" Jul 13 20:07:29.957136 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957081 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" containerName="maas-api" Jul 13 20:07:29.957291 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957174 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59db086-2f35-4625-b6c0-69613a619f83" containerName="revoke-keys" Jul 13 20:07:29.957291 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.957191 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b8e1de-9ae4-4493-b170-a96282d6cd7a" containerName="maas-api" Jul 13 20:07:29.961967 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.961946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:29.964945 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.964919 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:07:29.967882 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:29.967857 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv"] Jul 13 20:07:30.035578 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.035534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv\" (UID: \"312df522-7e67-4387-adbb-5b3c51af9243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:30.136156 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.136120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv\" (UID: \"312df522-7e67-4387-adbb-5b3c51af9243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:30.136790 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.136770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv\" (UID: \"312df522-7e67-4387-adbb-5b3c51af9243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:30.274486 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.274364 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:30.419387 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.419362 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv"] Jul 13 20:07:30.420564 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:07:30.420535 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312df522_7e67_4387_adbb_5b3c51af9243.slice/crio-e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb WatchSource:0}: Error finding container e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb: Status 404 returned error can't find the container with id e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb Jul 13 20:07:30.947081 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.946996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" event={"ID":"312df522-7e67-4387-adbb-5b3c51af9243","Type":"ContainerStarted","Data":"3cc101aafa9ec2f90e8296e2e739b9676e98b6ee099da97213e84c634b626e4a"} Jul 13 20:07:30.947081 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.947046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" event={"ID":"312df522-7e67-4387-adbb-5b3c51af9243","Type":"ContainerStarted","Data":"e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb"} Jul 13 20:07:30.965007 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:30.964954 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" podStartSLOduration=1.964940922 podStartE2EDuration="1.964940922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:07:30.963588204 +0000 UTC m=+2542.524986247" watchObservedRunningTime="2026-07-13 20:07:30.964940922 +0000 UTC m=+2542.526338965" Jul 13 20:07:31.953686 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:31.953647 2582 generic.go:356] "Generic (PLEG): container finished" podID="312df522-7e67-4387-adbb-5b3c51af9243" containerID="3cc101aafa9ec2f90e8296e2e739b9676e98b6ee099da97213e84c634b626e4a" exitCode=0 Jul 13 20:07:31.953834 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:31.953732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" event={"ID":"312df522-7e67-4387-adbb-5b3c51af9243","Type":"ContainerDied","Data":"3cc101aafa9ec2f90e8296e2e739b9676e98b6ee099da97213e84c634b626e4a"} Jul 13 20:07:33.097005 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.096965 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:33.160782 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.160750 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca\") pod \"312df522-7e67-4387-adbb-5b3c51af9243\" (UID: \"312df522-7e67-4387-adbb-5b3c51af9243\") " Jul 13 20:07:33.161131 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.161108 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "312df522-7e67-4387-adbb-5b3c51af9243" (UID: "312df522-7e67-4387-adbb-5b3c51af9243"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:07:33.262232 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.262144 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/312df522-7e67-4387-adbb-5b3c51af9243-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:33.967566 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.967541 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" Jul 13 20:07:33.967751 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.967541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv" event={"ID":"312df522-7e67-4387-adbb-5b3c51af9243","Type":"ContainerDied","Data":"e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb"} Jul 13 20:07:33.967751 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:33.967669 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e94ec63e8c7d7e5fc85bda51ceae5dd15324b06b2be06a3f8f7be628ca12a0fb" Jul 13 20:07:40.829460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:40.829408 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:07:40.829963 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:40.829693 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" podUID="c4c97f6f-c95b-4abd-893c-83a1c6880120" containerName="maas-api" containerID="cri-o://d066061cf0fde0c444dd608609e87f675b70d56498d80697d7c1cfeab82202aa" gracePeriod=30 Jul 13 20:07:41.012878 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.012808 2582 generic.go:356] "Generic (PLEG): container finished" podID="c4c97f6f-c95b-4abd-893c-83a1c6880120" containerID="d066061cf0fde0c444dd608609e87f675b70d56498d80697d7c1cfeab82202aa" exitCode=0 Jul 13 20:07:41.013056 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.013011 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" event={"ID":"c4c97f6f-c95b-4abd-893c-83a1c6880120","Type":"ContainerDied","Data":"d066061cf0fde0c444dd608609e87f675b70d56498d80697d7c1cfeab82202aa"} Jul 13 20:07:41.086018 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.085953 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:07:41.131021 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.130988 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca\") pod \"c4c97f6f-c95b-4abd-893c-83a1c6880120\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " Jul 13 20:07:41.131185 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.131043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb7lb\" (UniqueName: \"kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb\") pod \"c4c97f6f-c95b-4abd-893c-83a1c6880120\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " Jul 13 20:07:41.131185 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.131131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls\") pod \"c4c97f6f-c95b-4abd-893c-83a1c6880120\" (UID: \"c4c97f6f-c95b-4abd-893c-83a1c6880120\") " Jul 13 20:07:41.131373 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.131347 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4c97f6f-c95b-4abd-893c-83a1c6880120" (UID: "c4c97f6f-c95b-4abd-893c-83a1c6880120"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:07:41.131517 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.131500 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c97f6f-c95b-4abd-893c-83a1c6880120-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:41.133360 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.133327 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb" (OuterVolumeSpecName: "kube-api-access-vb7lb") pod "c4c97f6f-c95b-4abd-893c-83a1c6880120" (UID: "c4c97f6f-c95b-4abd-893c-83a1c6880120"). InnerVolumeSpecName "kube-api-access-vb7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:07:41.133491 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.133402 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4c97f6f-c95b-4abd-893c-83a1c6880120" (UID: "c4c97f6f-c95b-4abd-893c-83a1c6880120"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:07:41.232748 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.232717 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4c97f6f-c95b-4abd-893c-83a1c6880120-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:41.232748 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:41.232744 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7lb\" (UniqueName: \"kubernetes.io/projected/c4c97f6f-c95b-4abd-893c-83a1c6880120-kube-api-access-vb7lb\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:07:42.021358 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.021323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" event={"ID":"c4c97f6f-c95b-4abd-893c-83a1c6880120","Type":"ContainerDied","Data":"336285d8fce504f9b9a99aba11bdabe392ded41b82fd407f27061fd3f34f9a48"} Jul 13 20:07:42.021358 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.021349 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs" Jul 13 20:07:42.021888 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.021387 2582 scope.go:117] "RemoveContainer" containerID="d066061cf0fde0c444dd608609e87f675b70d56498d80697d7c1cfeab82202aa" Jul 13 20:07:42.041858 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.041823 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:07:42.047480 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.047454 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e52c5-5d8db94f47-vp7vs"] Jul 13 20:07:42.962071 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:42.962038 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c97f6f-c95b-4abd-893c-83a1c6880120" path="/var/lib/kubelet/pods/c4c97f6f-c95b-4abd-893c-83a1c6880120/volumes" Jul 13 20:07:59.713404 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.713292 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.713927 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312df522-7e67-4387-adbb-5b3c51af9243" containerName="revoke-keys" Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.713946 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="312df522-7e67-4387-adbb-5b3c51af9243" containerName="revoke-keys" Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.713967 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c97f6f-c95b-4abd-893c-83a1c6880120" containerName="maas-api" Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.713976 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c97f6f-c95b-4abd-893c-83a1c6880120" containerName="maas-api" Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.714088 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c97f6f-c95b-4abd-893c-83a1c6880120" containerName="maas-api" Jul 13 20:07:59.714388 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.714107 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="312df522-7e67-4387-adbb-5b3c51af9243" containerName="revoke-keys" Jul 13 20:07:59.721513 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.721010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.727883 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.727858 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-939152-serving-cert\"" Jul 13 20:07:59.728607 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.728579 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xhth\"" Jul 13 20:07:59.728723 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.728707 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 20:07:59.728926 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.728904 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 20:07:59.733785 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.733758 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:07:59.811788 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.811752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.811983 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.811803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh59c\" (UniqueName: \"kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.811983 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.811884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.912785 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.912751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.912985 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.912800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh59c\" (UniqueName: \"kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.912985 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.912832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.913369 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.913347 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.915400 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.915376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:07:59.923413 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:07:59.923384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh59c\" (UniqueName: \"kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c\") pod \"maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:08:00.040528 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:00.040436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:08:00.186155 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:00.186123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:08:00.186967 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:08:00.186939 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8609f46_4ce5_4193_9c95_429a2e6b9220.slice/crio-4e9d44c37ca354c2616bcf05eef8de1251dd4d726782fb2b5ae1525afa9ef6d1 WatchSource:0}: Error finding container 4e9d44c37ca354c2616bcf05eef8de1251dd4d726782fb2b5ae1525afa9ef6d1: Status 404 returned error can't find the container with id 4e9d44c37ca354c2616bcf05eef8de1251dd4d726782fb2b5ae1525afa9ef6d1 Jul 13 20:08:01.134010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:01.133976 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" event={"ID":"d8609f46-4ce5-4193-9c95-429a2e6b9220","Type":"ContainerStarted","Data":"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582"} Jul 13 20:08:01.134402 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:01.134032 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:08:01.134402 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:01.134049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" event={"ID":"d8609f46-4ce5-4193-9c95-429a2e6b9220","Type":"ContainerStarted","Data":"4e9d44c37ca354c2616bcf05eef8de1251dd4d726782fb2b5ae1525afa9ef6d1"} Jul 13 20:08:01.152622 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:01.152568 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" podStartSLOduration=1.7587099670000002 podStartE2EDuration="2.152550625s" totalImagesPullingTime="393.840658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:08:01.151087927 +0000 UTC m=+2572.712485987" watchObservedRunningTime="2026-07-13 20:08:01.152550625 +0000 UTC m=+2572.713948704" Jul 13 20:08:07.145933 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:07.145905 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:08:08.032893 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.032865 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:08:08.037616 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.037595 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.039828 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.039807 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-641463-serving-cert\"" Jul 13 20:08:08.044150 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.044126 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:08:08.192387 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.192351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.192770 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.192397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsmt\" (UniqueName: \"kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.192770 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.192519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.293527 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.293406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.293527 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.293484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsmt\" (UniqueName: \"kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.293784 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.293541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.294203 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.294174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.296076 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.296059 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.304214 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.304193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsmt\" (UniqueName: \"kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt\") pod \"maas-api-e2e-shared-b-641463-76777f6bb7-7g84j\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.350938 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.350891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:08.492507 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:08.492481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:08:08.493681 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:08:08.493658 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efc61dc_0ec6_4a2c_b51e_c750aad69bd7.slice/crio-27608f91e9dcc729be85869b4c6048a1dbdab18de81fcc361607bd64529474cc WatchSource:0}: Error finding container 27608f91e9dcc729be85869b4c6048a1dbdab18de81fcc361607bd64529474cc: Status 404 returned error can't find the container with id 27608f91e9dcc729be85869b4c6048a1dbdab18de81fcc361607bd64529474cc Jul 13 20:08:09.182060 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:09.182026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" event={"ID":"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7","Type":"ContainerStarted","Data":"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67"} Jul 13 20:08:09.182224 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:09.182078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" event={"ID":"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7","Type":"ContainerStarted","Data":"27608f91e9dcc729be85869b4c6048a1dbdab18de81fcc361607bd64529474cc"} Jul 13 20:08:09.182224 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:09.182131 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:08:09.199142 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:09.199080 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" podStartSLOduration=0.68278277 podStartE2EDuration="1.199058827s" totalImagesPullingTime="516.276057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:08:09.197759857 +0000 UTC m=+2580.759157919" watchObservedRunningTime="2026-07-13 20:08:09.199058827 +0000 UTC m=+2580.760456871" Jul 13 20:08:15.199086 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:08:15.199057 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:10:09.141098 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:10:09.141071 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:10:09.149949 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:10:09.149927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:11:49.089397 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.089304 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:11:49.092731 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.092710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.096532 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.096507 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-641463\"/\"openshift-service-ca.crt\"" Jul 13 20:11:49.096532 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.096514 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-641463\"/\"sub-test-model-641463-kserve-self-signed-certs\"" Jul 13 20:11:49.096744 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.096658 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-641463\"/\"default-dockercfg-tlzg5\"" Jul 13 20:11:49.096744 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.096692 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-641463\"/\"kube-root-ca.crt\"" Jul 13 20:11:49.104900 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.104882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:11:49.162603 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.162767 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.162767 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.162767 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkj9\" (UniqueName: \"kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.162767 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.162954 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.162793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263538 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263725 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263725 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263855 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263855 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.263855 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkj9\" (UniqueName: \"kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.264004 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263913 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.264004 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.263988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.264146 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.264128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.266010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.265985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.266264 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.266246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.273377 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.273347 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkj9\" (UniqueName: \"kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9\") pod \"sub-test-model-641463-kserve-6b7499ffc9-52rz4\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.403290 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.403257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:49.542815 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:49.542775 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:11:49.543458 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:11:49.543420 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d8a9d75_a8ae_449c_a726_9671b599b393.slice/crio-6c0ca526449683d4ab0497dcd6a01252ad0cfa3e9a31a0f069ecb78f295d5576 WatchSource:0}: Error finding container 6c0ca526449683d4ab0497dcd6a01252ad0cfa3e9a31a0f069ecb78f295d5576: Status 404 returned error can't find the container with id 6c0ca526449683d4ab0497dcd6a01252ad0cfa3e9a31a0f069ecb78f295d5576 Jul 13 20:11:50.462137 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:50.462087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" event={"ID":"4d8a9d75-a8ae-449c-a726-9671b599b393","Type":"ContainerStarted","Data":"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922"} Jul 13 20:11:50.462137 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:50.462143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" event={"ID":"4d8a9d75-a8ae-449c-a726-9671b599b393","Type":"ContainerStarted","Data":"6c0ca526449683d4ab0497dcd6a01252ad0cfa3e9a31a0f069ecb78f295d5576"} Jul 13 20:11:50.479163 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:50.479118 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" podStartSLOduration=1.479105355 podStartE2EDuration="1.479105355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:11:50.477900084 +0000 UTC m=+2802.039298130" watchObservedRunningTime="2026-07-13 20:11:50.479105355 +0000 UTC m=+2802.040503397" Jul 13 20:11:59.404473 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:59.404417 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:59.404873 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:59.404502 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:59.417466 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:59.417440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:11:59.530753 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:11:59.530726 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:12:53.137910 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:12:53.133734 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:13:08.158256 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.158153 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" podUID="4d8a9d75-a8ae-449c-a726-9671b599b393" containerName="main" containerID="cri-o://b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922" gracePeriod=45 Jul 13 20:13:08.410008 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.409945 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:13:08.569279 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569244 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569495 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569287 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569495 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569495 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569415 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vkj9\" (UniqueName: \"kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569495 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569707 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569502 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir\") pod \"4d8a9d75-a8ae-449c-a726-9671b599b393\" (UID: \"4d8a9d75-a8ae-449c-a726-9671b599b393\") " Jul 13 20:13:08.569707 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569551 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 20:13:08.569855 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569702 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home" (OuterVolumeSpecName: "home") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 20:13:08.569855 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569844 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-home\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.569928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569862 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-model-cache\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.569928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.569848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 20:13:08.571707 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.571684 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm" (OuterVolumeSpecName: "dshm") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 20:13:08.572062 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.572041 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:13:08.572171 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.572072 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9" (OuterVolumeSpecName: "kube-api-access-5vkj9") pod "4d8a9d75-a8ae-449c-a726-9671b599b393" (UID: "4d8a9d75-a8ae-449c-a726-9671b599b393"). InnerVolumeSpecName "kube-api-access-5vkj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:13:08.671279 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.671203 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8a9d75-a8ae-449c-a726-9671b599b393-tls-certs\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.671279 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.671232 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkj9\" (UniqueName: \"kubernetes.io/projected/4d8a9d75-a8ae-449c-a726-9671b599b393-kube-api-access-5vkj9\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.671279 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.671241 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-dshm\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.671279 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.671251 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8a9d75-a8ae-449c-a726-9671b599b393-tmp-dir\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:13:08.929808 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.929730 2582 generic.go:356] "Generic (PLEG): container finished" podID="4d8a9d75-a8ae-449c-a726-9671b599b393" containerID="b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922" exitCode=0 Jul 13 20:13:08.929808 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.929775 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" Jul 13 20:13:08.929983 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.929820 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" event={"ID":"4d8a9d75-a8ae-449c-a726-9671b599b393","Type":"ContainerDied","Data":"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922"} Jul 13 20:13:08.929983 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.929867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4" event={"ID":"4d8a9d75-a8ae-449c-a726-9671b599b393","Type":"ContainerDied","Data":"6c0ca526449683d4ab0497dcd6a01252ad0cfa3e9a31a0f069ecb78f295d5576"} Jul 13 20:13:08.929983 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.929888 2582 scope.go:117] "RemoveContainer" containerID="b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922" Jul 13 20:13:08.939709 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.939693 2582 scope.go:117] "RemoveContainer" containerID="b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922" Jul 13 20:13:08.939992 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:13:08.939972 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922\": container with ID starting with b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922 not found: ID does not exist" containerID="b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922" Jul 13 20:13:08.940069 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.939999 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922"} err="failed to get container status \"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922\": rpc error: code = NotFound desc = could not find container \"b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922\": container with ID starting with b358d93eae8990a246c430973fff4e2e327b21d652015de20a2a4090903fb922 not found: ID does not exist" Jul 13 20:13:08.954741 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.954722 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:13:08.961968 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:08.961946 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-641463/sub-test-model-641463-kserve-6b7499ffc9-52rz4"] Jul 13 20:13:10.961418 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:13:10.961381 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8a9d75-a8ae-449c-a726-9671b599b393" path="/var/lib/kubelet/pods/4d8a9d75-a8ae-449c-a726-9671b599b393/volumes" Jul 13 20:14:19.975114 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.975078 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-139940049987760"] Jul 13 20:14:19.975562 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.975501 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8a9d75-a8ae-449c-a726-9671b599b393" containerName="main" Jul 13 20:14:19.975562 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.975513 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8a9d75-a8ae-449c-a726-9671b599b393" containerName="main" Jul 13 20:14:19.975640 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.975584 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8a9d75-a8ae-449c-a726-9671b599b393" containerName="main" Jul 13 20:14:19.978665 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.978650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:19.981556 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.981527 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pt8zs\"" Jul 13 20:14:19.984257 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:19.984233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940049987760"] Jul 13 20:14:20.009076 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.009046 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmqh\" (UniqueName: \"kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh\") pod \"test-curl-7481-139940049987760\" (UID: \"83bdb28a-9797-41f6-a6c4-cb2b86a9e224\") " pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:20.110188 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.110151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmqh\" (UniqueName: \"kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh\") pod \"test-curl-7481-139940049987760\" (UID: \"83bdb28a-9797-41f6-a6c4-cb2b86a9e224\") " pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:20.120371 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.120335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmqh\" (UniqueName: \"kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh\") pod \"test-curl-7481-139940049987760\" (UID: \"83bdb28a-9797-41f6-a6c4-cb2b86a9e224\") " pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:20.288712 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.288617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:20.425376 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.425337 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940049987760"] Jul 13 20:14:20.426659 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:20.426630 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83bdb28a_9797_41f6_a6c4_cb2b86a9e224.slice/crio-eb6a5e26f2ad35f84d32e79dc1c3a5385cad85c93bba03c7f9c86bc31b3bd1a5 WatchSource:0}: Error finding container eb6a5e26f2ad35f84d32e79dc1c3a5385cad85c93bba03c7f9c86bc31b3bd1a5: Status 404 returned error can't find the container with id eb6a5e26f2ad35f84d32e79dc1c3a5385cad85c93bba03c7f9c86bc31b3bd1a5 Jul 13 20:14:20.427988 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:20.427970 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 20:14:21.364034 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:21.364005 2582 generic.go:356] "Generic (PLEG): container finished" podID="83bdb28a-9797-41f6-a6c4-cb2b86a9e224" containerID="cc94867156336f7031affdfcab167757e94e39f6f4d3c6dfe8e0dfce831e18e3" exitCode=0 Jul 13 20:14:21.364460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:21.364089 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940049987760" event={"ID":"83bdb28a-9797-41f6-a6c4-cb2b86a9e224","Type":"ContainerDied","Data":"cc94867156336f7031affdfcab167757e94e39f6f4d3c6dfe8e0dfce831e18e3"} Jul 13 20:14:21.364460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:21.364131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940049987760" event={"ID":"83bdb28a-9797-41f6-a6c4-cb2b86a9e224","Type":"ContainerStarted","Data":"eb6a5e26f2ad35f84d32e79dc1c3a5385cad85c93bba03c7f9c86bc31b3bd1a5"} Jul 13 20:14:22.508072 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.508045 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:22.534190 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.534163 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tmqh\" (UniqueName: \"kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh\") pod \"83bdb28a-9797-41f6-a6c4-cb2b86a9e224\" (UID: \"83bdb28a-9797-41f6-a6c4-cb2b86a9e224\") " Jul 13 20:14:22.536399 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.536369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh" (OuterVolumeSpecName: "kube-api-access-6tmqh") pod "83bdb28a-9797-41f6-a6c4-cb2b86a9e224" (UID: "83bdb28a-9797-41f6-a6c4-cb2b86a9e224"). InnerVolumeSpecName "kube-api-access-6tmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:22.545465 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.545414 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-139940049987760_83bdb28a-9797-41f6-a6c4-cb2b86a9e224/test-curl-7481-139940049987760/0.log" Jul 13 20:14:22.572235 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.572202 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-139940049987760"] Jul 13 20:14:22.575007 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.574985 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-139940049987760"] Jul 13 20:14:22.635538 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.635460 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmqh\" (UniqueName: \"kubernetes.io/projected/83bdb28a-9797-41f6-a6c4-cb2b86a9e224-kube-api-access-6tmqh\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:22.699477 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.699437 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-139940044269488"] Jul 13 20:14:22.699871 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.699856 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bdb28a-9797-41f6-a6c4-cb2b86a9e224" containerName="test-curl-7481-139940049987760" Jul 13 20:14:22.699913 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.699872 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bdb28a-9797-41f6-a6c4-cb2b86a9e224" containerName="test-curl-7481-139940049987760" Jul 13 20:14:22.699948 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.699937 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bdb28a-9797-41f6-a6c4-cb2b86a9e224" containerName="test-curl-7481-139940049987760" Jul 13 20:14:22.703021 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.703004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:22.709858 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.709832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044269488"] Jul 13 20:14:22.736659 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.736625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjh2\" (UniqueName: \"kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2\") pod \"test-curl-7481-139940044269488\" (UID: \"e902314b-4a5d-4334-8c77-82efe31494d0\") " pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:22.837600 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.837566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjh2\" (UniqueName: \"kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2\") pod \"test-curl-7481-139940044269488\" (UID: \"e902314b-4a5d-4334-8c77-82efe31494d0\") " pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:22.846452 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.846392 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjh2\" (UniqueName: \"kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2\") pod \"test-curl-7481-139940044269488\" (UID: \"e902314b-4a5d-4334-8c77-82efe31494d0\") " pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:22.961576 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:22.961496 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83bdb28a-9797-41f6-a6c4-cb2b86a9e224" path="/var/lib/kubelet/pods/83bdb28a-9797-41f6-a6c4-cb2b86a9e224/volumes" Jul 13 20:14:23.013402 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:23.013366 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:23.151786 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:23.151762 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044269488"] Jul 13 20:14:23.153162 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:23.153133 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode902314b_4a5d_4334_8c77_82efe31494d0.slice/crio-e673af63c989f98c14cde9ac69c5d197aa9aae82bc6a0aaedcef1a3e971d6577 WatchSource:0}: Error finding container e673af63c989f98c14cde9ac69c5d197aa9aae82bc6a0aaedcef1a3e971d6577: Status 404 returned error can't find the container with id e673af63c989f98c14cde9ac69c5d197aa9aae82bc6a0aaedcef1a3e971d6577 Jul 13 20:14:23.377177 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:23.377145 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044269488" event={"ID":"e902314b-4a5d-4334-8c77-82efe31494d0","Type":"ContainerStarted","Data":"e673af63c989f98c14cde9ac69c5d197aa9aae82bc6a0aaedcef1a3e971d6577"} Jul 13 20:14:23.378372 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:23.378351 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940049987760" Jul 13 20:14:23.378503 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:23.378404 2582 scope.go:117] "RemoveContainer" containerID="cc94867156336f7031affdfcab167757e94e39f6f4d3c6dfe8e0dfce831e18e3" Jul 13 20:14:24.386440 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:24.386397 2582 generic.go:356] "Generic (PLEG): container finished" podID="e902314b-4a5d-4334-8c77-82efe31494d0" containerID="41a81ed901db56b0d74457a95637f311b2f4c39ecee14ff96bcf9d20e5eda71c" exitCode=0 Jul 13 20:14:24.386812 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:24.386467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044269488" event={"ID":"e902314b-4a5d-4334-8c77-82efe31494d0","Type":"ContainerDied","Data":"41a81ed901db56b0d74457a95637f311b2f4c39ecee14ff96bcf9d20e5eda71c"} Jul 13 20:14:25.527036 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.527009 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:25.548126 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.548091 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-139940044269488_e902314b-4a5d-4334-8c77-82efe31494d0/test-curl-7481-139940044269488/0.log" Jul 13 20:14:25.564601 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.564575 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxjh2\" (UniqueName: \"kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2\") pod \"e902314b-4a5d-4334-8c77-82efe31494d0\" (UID: \"e902314b-4a5d-4334-8c77-82efe31494d0\") " Jul 13 20:14:25.566732 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.566707 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2" (OuterVolumeSpecName: "kube-api-access-xxjh2") pod "e902314b-4a5d-4334-8c77-82efe31494d0" (UID: "e902314b-4a5d-4334-8c77-82efe31494d0"). InnerVolumeSpecName "kube-api-access-xxjh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:25.573483 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.573464 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-139940044269488"] Jul 13 20:14:25.578928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.578909 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-139940044269488"] Jul 13 20:14:25.665801 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.665732 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjh2\" (UniqueName: \"kubernetes.io/projected/e902314b-4a5d-4334-8c77-82efe31494d0-kube-api-access-xxjh2\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:25.745184 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.745154 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:25.745598 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.745584 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e902314b-4a5d-4334-8c77-82efe31494d0" containerName="test-curl-7481-139940044269488" Jul 13 20:14:25.745649 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.745599 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e902314b-4a5d-4334-8c77-82efe31494d0" containerName="test-curl-7481-139940044269488" Jul 13 20:14:25.745681 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.745665 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e902314b-4a5d-4334-8c77-82efe31494d0" containerName="test-curl-7481-139940044269488" Jul 13 20:14:25.748786 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.748772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:25.754873 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.754853 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:25.766118 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.766093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzlk\" (UniqueName: \"kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk\") pod \"test-curl-7481-139940044402736\" (UID: \"470d35fb-7180-41a5-85fa-f6a1a1434864\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:25.867124 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.867087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzlk\" (UniqueName: \"kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk\") pod \"test-curl-7481-139940044402736\" (UID: \"470d35fb-7180-41a5-85fa-f6a1a1434864\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:25.875712 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:25.875681 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzlk\" (UniqueName: \"kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk\") pod \"test-curl-7481-139940044402736\" (UID: \"470d35fb-7180-41a5-85fa-f6a1a1434864\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:26.059415 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.059336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:26.196471 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.196442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:26.197030 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:26.197003 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod470d35fb_7180_41a5_85fa_f6a1a1434864.slice/crio-1229c10f2a6bb87a737c4ec9c9254fcf2fac2fed05a4db0b03427f899dcc7e27 WatchSource:0}: Error finding container 1229c10f2a6bb87a737c4ec9c9254fcf2fac2fed05a4db0b03427f899dcc7e27: Status 404 returned error can't find the container with id 1229c10f2a6bb87a737c4ec9c9254fcf2fac2fed05a4db0b03427f899dcc7e27 Jul 13 20:14:26.399644 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.399617 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:26.399829 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.399656 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e673af63c989f98c14cde9ac69c5d197aa9aae82bc6a0aaedcef1a3e971d6577" Jul 13 20:14:26.400968 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.400943 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044402736" event={"ID":"470d35fb-7180-41a5-85fa-f6a1a1434864","Type":"ContainerStarted","Data":"1229c10f2a6bb87a737c4ec9c9254fcf2fac2fed05a4db0b03427f899dcc7e27"} Jul 13 20:14:26.410020 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:14:26.409998 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7481-139940044269488\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="e902314b-4a5d-4334-8c77-82efe31494d0" pod="opendatahub/test-curl-7481-139940044269488" Jul 13 20:14:26.961289 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:26.961248 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e902314b-4a5d-4334-8c77-82efe31494d0" path="/var/lib/kubelet/pods/e902314b-4a5d-4334-8c77-82efe31494d0/volumes" Jul 13 20:14:27.408217 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:27.408183 2582 generic.go:356] "Generic (PLEG): container finished" podID="470d35fb-7180-41a5-85fa-f6a1a1434864" containerID="f43ab50e735b8044a95ba5704c338ce1358a6c7a4c0d3f7e832810acfc4c922d" exitCode=0 Jul 13 20:14:27.408416 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:27.408265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044402736" event={"ID":"470d35fb-7180-41a5-85fa-f6a1a1434864","Type":"ContainerDied","Data":"f43ab50e735b8044a95ba5704c338ce1358a6c7a4c0d3f7e832810acfc4c922d"} Jul 13 20:14:28.549071 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.549041 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:28.569108 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.569077 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-139940044402736_470d35fb-7180-41a5-85fa-f6a1a1434864/test-curl-7481-139940044402736/0.log" Jul 13 20:14:28.590308 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.590278 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plzlk\" (UniqueName: \"kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk\") pod \"470d35fb-7180-41a5-85fa-f6a1a1434864\" (UID: \"470d35fb-7180-41a5-85fa-f6a1a1434864\") " Jul 13 20:14:28.592844 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.592816 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk" (OuterVolumeSpecName: "kube-api-access-plzlk") pod "470d35fb-7180-41a5-85fa-f6a1a1434864" (UID: "470d35fb-7180-41a5-85fa-f6a1a1434864"). InnerVolumeSpecName "kube-api-access-plzlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:28.594353 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.594332 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:28.599988 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.599968 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:28.691956 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.691865 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzlk\" (UniqueName: \"kubernetes.io/projected/470d35fb-7180-41a5-85fa-f6a1a1434864-kube-api-access-plzlk\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:28.719853 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.719825 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:28.720234 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.720222 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470d35fb-7180-41a5-85fa-f6a1a1434864" containerName="test-curl-7481-139940044402736" Jul 13 20:14:28.720272 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.720236 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="470d35fb-7180-41a5-85fa-f6a1a1434864" containerName="test-curl-7481-139940044402736" Jul 13 20:14:28.720324 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.720315 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="470d35fb-7180-41a5-85fa-f6a1a1434864" containerName="test-curl-7481-139940044402736" Jul 13 20:14:28.723253 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.723237 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:28.727612 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.727588 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:28.792307 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.792278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4hj\" (UniqueName: \"kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj\") pod \"test-curl-7481-139940044402736\" (UID: \"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:28.892913 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.892882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4hj\" (UniqueName: \"kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj\") pod \"test-curl-7481-139940044402736\" (UID: \"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:28.901413 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.901385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4hj\" (UniqueName: \"kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj\") pod \"test-curl-7481-139940044402736\" (UID: \"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a\") " pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:28.969003 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:28.968924 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="470d35fb-7180-41a5-85fa-f6a1a1434864" path="/var/lib/kubelet/pods/470d35fb-7180-41a5-85fa-f6a1a1434864/volumes" Jul 13 20:14:29.033926 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:29.033893 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:29.167839 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:29.167812 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:29.169504 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:29.169480 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf03c2dd9_a835_4f21_97a0_f2f42b8e8b9a.slice/crio-6169b87d153f19f43d289cd1eb46f24ee80b76c7cd976dbe7fd51680b3d6d558 WatchSource:0}: Error finding container 6169b87d153f19f43d289cd1eb46f24ee80b76c7cd976dbe7fd51680b3d6d558: Status 404 returned error can't find the container with id 6169b87d153f19f43d289cd1eb46f24ee80b76c7cd976dbe7fd51680b3d6d558 Jul 13 20:14:29.420126 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:29.420092 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:29.420315 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:29.420135 2582 scope.go:117] "RemoveContainer" containerID="f43ab50e735b8044a95ba5704c338ce1358a6c7a4c0d3f7e832810acfc4c922d" Jul 13 20:14:29.421551 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:29.421361 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044402736" event={"ID":"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a","Type":"ContainerStarted","Data":"6169b87d153f19f43d289cd1eb46f24ee80b76c7cd976dbe7fd51680b3d6d558"} Jul 13 20:14:30.429032 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:30.428997 2582 generic.go:356] "Generic (PLEG): container finished" podID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" containerID="f994ae71ae343e9232dfb69b135f6a5bd77111f6e3651834e0d3a82adab83ad5" exitCode=0 Jul 13 20:14:30.429505 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:30.429096 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044402736" event={"ID":"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a","Type":"ContainerDied","Data":"f994ae71ae343e9232dfb69b135f6a5bd77111f6e3651834e0d3a82adab83ad5"} Jul 13 20:14:31.571104 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.571081 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:31.591298 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.591274 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-139940044402736_f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a/test-curl-7481-139940044402736/0.log" Jul 13 20:14:31.615545 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.615515 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:31.617852 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.617823 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr4hj\" (UniqueName: \"kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj\") pod \"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a\" (UID: \"f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a\") " Jul 13 20:14:31.620270 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.620246 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj" (OuterVolumeSpecName: "kube-api-access-xr4hj") pod "f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" (UID: "f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a"). InnerVolumeSpecName "kube-api-access-xr4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:31.620789 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.620771 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-139940044402736"] Jul 13 20:14:31.719119 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.719038 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr4hj\" (UniqueName: \"kubernetes.io/projected/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a-kube-api-access-xr4hj\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:31.746274 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.746248 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-139940044403744"] Jul 13 20:14:31.746703 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.746689 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" containerName="test-curl-7481-139940044402736" Jul 13 20:14:31.746746 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.746706 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" containerName="test-curl-7481-139940044402736" Jul 13 20:14:31.746817 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.746805 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" containerName="test-curl-7481-139940044402736" Jul 13 20:14:31.749936 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.749920 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:31.756402 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.756377 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044403744"] Jul 13 20:14:31.820198 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.820165 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djz4h\" (UniqueName: \"kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h\") pod \"test-curl-7481-139940044403744\" (UID: \"ac46e2bc-4729-4a36-b02e-3ed42754c64e\") " pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:31.921293 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.921263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djz4h\" (UniqueName: \"kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h\") pod \"test-curl-7481-139940044403744\" (UID: \"ac46e2bc-4729-4a36-b02e-3ed42754c64e\") " pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:31.929019 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:31.928994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djz4h\" (UniqueName: \"kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h\") pod \"test-curl-7481-139940044403744\" (UID: \"ac46e2bc-4729-4a36-b02e-3ed42754c64e\") " pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:32.061192 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.061113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:32.197896 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.197867 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-139940044403744"] Jul 13 20:14:32.198849 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:32.198828 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac46e2bc_4729_4a36_b02e_3ed42754c64e.slice/crio-1993adba3edf19d2ece21e1baf268be07bf0df478c4e7c09e38f5461fffa1374 WatchSource:0}: Error finding container 1993adba3edf19d2ece21e1baf268be07bf0df478c4e7c09e38f5461fffa1374: Status 404 returned error can't find the container with id 1993adba3edf19d2ece21e1baf268be07bf0df478c4e7c09e38f5461fffa1374 Jul 13 20:14:32.441357 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.441329 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:32.441357 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.441360 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6169b87d153f19f43d289cd1eb46f24ee80b76c7cd976dbe7fd51680b3d6d558" Jul 13 20:14:32.442583 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.442558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044403744" event={"ID":"ac46e2bc-4729-4a36-b02e-3ed42754c64e","Type":"ContainerStarted","Data":"1993adba3edf19d2ece21e1baf268be07bf0df478c4e7c09e38f5461fffa1374"} Jul 13 20:14:32.451730 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:14:32.451707 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7481-139940044402736\" is forbidden: User \"system:node:ip-10-0-135-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-83.ec2.internal' and this object" podUID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" pod="opendatahub/test-curl-7481-139940044402736" Jul 13 20:14:32.961460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:32.961411 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a" path="/var/lib/kubelet/pods/f03c2dd9-a835-4f21-97a0-f2f42b8e8b9a/volumes" Jul 13 20:14:33.450111 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:33.450076 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac46e2bc-4729-4a36-b02e-3ed42754c64e" containerID="3287a76ecbfa9bbc7df4aad66551211369d973f90e668e33698dda5a3ea2ecab" exitCode=0 Jul 13 20:14:33.450286 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:33.450165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-139940044403744" event={"ID":"ac46e2bc-4729-4a36-b02e-3ed42754c64e","Type":"ContainerDied","Data":"3287a76ecbfa9bbc7df4aad66551211369d973f90e668e33698dda5a3ea2ecab"} Jul 13 20:14:34.597588 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.597568 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:34.619253 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.619227 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7481-139940044403744_ac46e2bc-4729-4a36-b02e-3ed42754c64e/test-curl-7481-139940044403744/0.log" Jul 13 20:14:34.645094 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.645061 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djz4h\" (UniqueName: \"kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h\") pod \"ac46e2bc-4729-4a36-b02e-3ed42754c64e\" (UID: \"ac46e2bc-4729-4a36-b02e-3ed42754c64e\") " Jul 13 20:14:34.645732 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.645708 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-139940044403744"] Jul 13 20:14:34.647594 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.647567 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h" (OuterVolumeSpecName: "kube-api-access-djz4h") pod "ac46e2bc-4729-4a36-b02e-3ed42754c64e" (UID: "ac46e2bc-4729-4a36-b02e-3ed42754c64e"). InnerVolumeSpecName "kube-api-access-djz4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:34.649844 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.649827 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-139940044403744"] Jul 13 20:14:34.746082 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.746006 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djz4h\" (UniqueName: \"kubernetes.io/projected/ac46e2bc-4729-4a36-b02e-3ed42754c64e-kube-api-access-djz4h\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:34.961811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:34.961780 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac46e2bc-4729-4a36-b02e-3ed42754c64e" path="/var/lib/kubelet/pods/ac46e2bc-4729-4a36-b02e-3ed42754c64e/volumes" Jul 13 20:14:35.464621 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:35.464596 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-139940044403744" Jul 13 20:14:35.464621 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:35.464608 2582 scope.go:117] "RemoveContainer" containerID="3287a76ecbfa9bbc7df4aad66551211369d973f90e668e33698dda5a3ea2ecab" Jul 13 20:14:38.258040 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.257997 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk"] Jul 13 20:14:38.258516 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.258413 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac46e2bc-4729-4a36-b02e-3ed42754c64e" containerName="test-curl-7481-139940044403744" Jul 13 20:14:38.258516 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.258438 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac46e2bc-4729-4a36-b02e-3ed42754c64e" containerName="test-curl-7481-139940044403744" Jul 13 20:14:38.258608 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.258527 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac46e2bc-4729-4a36-b02e-3ed42754c64e" containerName="test-curl-7481-139940044403744" Jul 13 20:14:38.262953 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.262935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:38.265781 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.265753 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:14:38.269581 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.269555 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk"] Jul 13 20:14:38.376968 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.376933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-939152-4bgbk\" (UID: \"1106abd1-1755-4589-b624-970be427ec72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:38.477587 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.477552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-939152-4bgbk\" (UID: \"1106abd1-1755-4589-b624-970be427ec72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:38.478139 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.478121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-939152-4bgbk\" (UID: \"1106abd1-1755-4589-b624-970be427ec72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:38.576294 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.576221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:38.737847 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:38.737818 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk"] Jul 13 20:14:38.739998 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:14:38.739958 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1106abd1_1755_4589_b624_970be427ec72.slice/crio-c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607 WatchSource:0}: Error finding container c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607: Status 404 returned error can't find the container with id c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607 Jul 13 20:14:39.490820 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:39.490792 2582 generic.go:356] "Generic (PLEG): container finished" podID="1106abd1-1755-4589-b624-970be427ec72" containerID="d2758fb6e6e5767b90d2db9f4295cfbd0ec1465cc32599607d9d20113abad4a8" exitCode=0 Jul 13 20:14:39.491149 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:39.490859 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" event={"ID":"1106abd1-1755-4589-b624-970be427ec72","Type":"ContainerDied","Data":"d2758fb6e6e5767b90d2db9f4295cfbd0ec1465cc32599607d9d20113abad4a8"} Jul 13 20:14:39.491149 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:39.490933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" event={"ID":"1106abd1-1755-4589-b624-970be427ec72","Type":"ContainerStarted","Data":"c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607"} Jul 13 20:14:40.635687 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:40.635662 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:40.705588 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:40.705556 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca\") pod \"1106abd1-1755-4589-b624-970be427ec72\" (UID: \"1106abd1-1755-4589-b624-970be427ec72\") " Jul 13 20:14:40.705896 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:40.705874 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1106abd1-1755-4589-b624-970be427ec72" (UID: "1106abd1-1755-4589-b624-970be427ec72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:14:40.807241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:40.807164 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1106abd1-1755-4589-b624-970be427ec72-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:41.504128 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:41.504099 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" Jul 13 20:14:41.504128 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:41.504104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-939152-4bgbk" event={"ID":"1106abd1-1755-4589-b624-970be427ec72","Type":"ContainerDied","Data":"c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607"} Jul 13 20:14:41.504335 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:41.504149 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c205bd26a961ecc8e12f711a4320ce3776b617e2321126d0212f4ccdadd3f607" Jul 13 20:14:48.684869 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:48.684831 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:14:48.685498 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:48.685082 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" podUID="d8609f46-4ce5-4193-9c95-429a2e6b9220" containerName="maas-api" containerID="cri-o://c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582" gracePeriod=30 Jul 13 20:14:48.928714 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:48.928678 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:14:49.082145 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.082059 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls\") pod \"d8609f46-4ce5-4193-9c95-429a2e6b9220\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " Jul 13 20:14:49.082145 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.082116 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh59c\" (UniqueName: \"kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c\") pod \"d8609f46-4ce5-4193-9c95-429a2e6b9220\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " Jul 13 20:14:49.082378 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.082169 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca\") pod \"d8609f46-4ce5-4193-9c95-429a2e6b9220\" (UID: \"d8609f46-4ce5-4193-9c95-429a2e6b9220\") " Jul 13 20:14:49.082682 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.082652 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8609f46-4ce5-4193-9c95-429a2e6b9220" (UID: "d8609f46-4ce5-4193-9c95-429a2e6b9220"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:14:49.084406 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.084385 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8609f46-4ce5-4193-9c95-429a2e6b9220" (UID: "d8609f46-4ce5-4193-9c95-429a2e6b9220"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:14:49.084515 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.084494 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c" (OuterVolumeSpecName: "kube-api-access-wh59c") pod "d8609f46-4ce5-4193-9c95-429a2e6b9220" (UID: "d8609f46-4ce5-4193-9c95-429a2e6b9220"). InnerVolumeSpecName "kube-api-access-wh59c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:14:49.183444 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.183399 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8609f46-4ce5-4193-9c95-429a2e6b9220-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:49.183444 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.183445 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh59c\" (UniqueName: \"kubernetes.io/projected/d8609f46-4ce5-4193-9c95-429a2e6b9220-kube-api-access-wh59c\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:49.183628 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.183457 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8609f46-4ce5-4193-9c95-429a2e6b9220-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:14:49.555500 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.555466 2582 generic.go:356] "Generic (PLEG): container finished" podID="d8609f46-4ce5-4193-9c95-429a2e6b9220" containerID="c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582" exitCode=0 Jul 13 20:14:49.555671 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.555513 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" Jul 13 20:14:49.555671 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.555519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" event={"ID":"d8609f46-4ce5-4193-9c95-429a2e6b9220","Type":"ContainerDied","Data":"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582"} Jul 13 20:14:49.555671 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.555560 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z" event={"ID":"d8609f46-4ce5-4193-9c95-429a2e6b9220","Type":"ContainerDied","Data":"4e9d44c37ca354c2616bcf05eef8de1251dd4d726782fb2b5ae1525afa9ef6d1"} Jul 13 20:14:49.555671 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.555581 2582 scope.go:117] "RemoveContainer" containerID="c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582" Jul 13 20:14:49.565241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.565223 2582 scope.go:117] "RemoveContainer" containerID="c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582" Jul 13 20:14:49.565643 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:14:49.565623 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582\": container with ID starting with c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582 not found: ID does not exist" containerID="c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582" Jul 13 20:14:49.565700 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.565652 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582"} err="failed to get container status \"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582\": rpc error: code = NotFound desc = could not find container \"c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582\": container with ID starting with c8c59d151e3b389f23c196eae90b27ba105849106b17595ee46be59614804582 not found: ID does not exist" Jul 13 20:14:49.579036 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.579013 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:14:49.582943 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:49.582921 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-939152-7d8567fc97-qpd2z"] Jul 13 20:14:50.961446 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:14:50.961403 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8609f46-4ce5-4193-9c95-429a2e6b9220" path="/var/lib/kubelet/pods/d8609f46-4ce5-4193-9c95-429a2e6b9220/volumes" Jul 13 20:15:00.148948 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.148912 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f"] Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149483 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1106abd1-1755-4589-b624-970be427ec72" containerName="revoke-keys" Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149499 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1106abd1-1755-4589-b624-970be427ec72" containerName="revoke-keys" Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149510 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8609f46-4ce5-4193-9c95-429a2e6b9220" containerName="maas-api" Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149518 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8609f46-4ce5-4193-9c95-429a2e6b9220" containerName="maas-api" Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149601 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8609f46-4ce5-4193-9c95-429a2e6b9220" containerName="maas-api" Jul 13 20:15:00.151362 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.149625 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1106abd1-1755-4589-b624-970be427ec72" containerName="revoke-keys" Jul 13 20:15:00.152715 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.152697 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc"] Jul 13 20:15:00.152816 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.152802 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" Jul 13 20:15:00.155243 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.155222 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:15:00.156326 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.156310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" Jul 13 20:15:00.161106 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.161066 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f"] Jul 13 20:15:00.165547 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.165529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" Jul 13 20:15:00.172284 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.172263 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc"] Jul 13 20:15:00.176452 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.176264 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" Jul 13 20:15:00.525928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.525898 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f"] Jul 13 20:15:00.526100 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:15:00.526052 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9928cdb_f826_49e8_925a_5dd4c22db0d1.slice/crio-2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9 WatchSource:0}: Error finding container 2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9: Status 404 returned error can't find the container with id 2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9 Jul 13 20:15:00.554165 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.554139 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc"] Jul 13 20:15:00.555163 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:15:00.555140 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48443d4_367c_4213_984d_f9e618d2daff.slice/crio-fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c WatchSource:0}: Error finding container fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c: Status 404 returned error can't find the container with id fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c Jul 13 20:15:00.629369 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.629331 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" event={"ID":"b48443d4-367c-4213-984d-f9e618d2daff","Type":"ContainerStarted","Data":"fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c"} Jul 13 20:15:00.630608 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:00.630575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" event={"ID":"e9928cdb-f826-49e8-925a-5dd4c22db0d1","Type":"ContainerStarted","Data":"2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9"} Jul 13 20:15:01.638548 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:01.638505 2582 generic.go:356] "Generic (PLEG): container finished" podID="e9928cdb-f826-49e8-925a-5dd4c22db0d1" containerID="8d39be9fbc0af7d565ae3cce45ac08148f0ff034ee8d37118eaff6598133b890" exitCode=0 Jul 13 20:15:01.638989 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:01.638608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" event={"ID":"e9928cdb-f826-49e8-925a-5dd4c22db0d1","Type":"ContainerDied","Data":"8d39be9fbc0af7d565ae3cce45ac08148f0ff034ee8d37118eaff6598133b890"} Jul 13 20:15:01.640276 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:01.640255 2582 generic.go:356] "Generic (PLEG): container finished" podID="b48443d4-367c-4213-984d-f9e618d2daff" containerID="4fbd82a25a9fc6e07009faf3edc1feaf0e799f6afb2152289a070cf534d9fb7b" exitCode=0 Jul 13 20:15:01.640341 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:01.640316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" event={"ID":"b48443d4-367c-4213-984d-f9e618d2daff","Type":"ContainerDied","Data":"4fbd82a25a9fc6e07009faf3edc1feaf0e799f6afb2152289a070cf534d9fb7b"} Jul 13 20:15:02.800572 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:02.800542 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" Jul 13 20:15:02.804297 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:02.804278 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" Jul 13 20:15:03.653912 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.653883 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" Jul 13 20:15:03.654115 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.653921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc" event={"ID":"b48443d4-367c-4213-984d-f9e618d2daff","Type":"ContainerDied","Data":"fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c"} Jul 13 20:15:03.654115 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.653981 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0cb2dbab50b386d7c89b00cddd1f9db729b4e10dcc85736b85a9e93a1cbe3c" Jul 13 20:15:03.655304 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.655277 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" Jul 13 20:15:03.655304 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.655293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732895-8986f" event={"ID":"e9928cdb-f826-49e8-925a-5dd4c22db0d1","Type":"ContainerDied","Data":"2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9"} Jul 13 20:15:03.655494 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:03.655329 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4569f98a8a35076ad46c001e114edb11367873100a986a10f3bc576aca39e9" Jul 13 20:15:04.476995 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.476962 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc"] Jul 13 20:15:04.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477375 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9928cdb-f826-49e8-925a-5dd4c22db0d1" containerName="cleanup" Jul 13 20:15:04.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477386 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9928cdb-f826-49e8-925a-5dd4c22db0d1" containerName="cleanup" Jul 13 20:15:04.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477399 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48443d4-367c-4213-984d-f9e618d2daff" containerName="cleanup" Jul 13 20:15:04.477460 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477405 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48443d4-367c-4213-984d-f9e618d2daff" containerName="cleanup" Jul 13 20:15:04.477614 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477473 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9928cdb-f826-49e8-925a-5dd4c22db0d1" containerName="cleanup" Jul 13 20:15:04.477614 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.477483 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48443d4-367c-4213-984d-f9e618d2daff" containerName="cleanup" Jul 13 20:15:04.480739 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.480723 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:04.483485 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.483465 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hk9td\"" Jul 13 20:15:04.490620 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.490598 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc"] Jul 13 20:15:04.516685 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.516649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-641463-pcvjc\" (UID: \"6d250128-4d32-4bfa-bf99-f5566007a147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:04.617811 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.617770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-641463-pcvjc\" (UID: \"6d250128-4d32-4bfa-bf99-f5566007a147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:04.618405 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.618379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-641463-pcvjc\" (UID: \"6d250128-4d32-4bfa-bf99-f5566007a147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:04.792637 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.792546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:04.939481 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:15:04.939449 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d250128_4d32_4bfa_bf99_f5566007a147.slice/crio-221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541 WatchSource:0}: Error finding container 221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541: Status 404 returned error can't find the container with id 221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541 Jul 13 20:15:04.939597 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:04.939548 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc"] Jul 13 20:15:05.669175 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:05.669142 2582 generic.go:356] "Generic (PLEG): container finished" podID="6d250128-4d32-4bfa-bf99-f5566007a147" containerID="cca539cbfe2054596d4c5f9a0a6bb6590d45b22d1815b6474df126bc7b8e8244" exitCode=0 Jul 13 20:15:05.669550 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:05.669228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" event={"ID":"6d250128-4d32-4bfa-bf99-f5566007a147","Type":"ContainerDied","Data":"cca539cbfe2054596d4c5f9a0a6bb6590d45b22d1815b6474df126bc7b8e8244"} Jul 13 20:15:05.669550 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:05.669282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" event={"ID":"6d250128-4d32-4bfa-bf99-f5566007a147","Type":"ContainerStarted","Data":"221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541"} Jul 13 20:15:06.814373 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:06.814349 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:06.835735 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:06.835708 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca\") pod \"6d250128-4d32-4bfa-bf99-f5566007a147\" (UID: \"6d250128-4d32-4bfa-bf99-f5566007a147\") " Jul 13 20:15:06.836094 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:06.836069 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d250128-4d32-4bfa-bf99-f5566007a147" (UID: "6d250128-4d32-4bfa-bf99-f5566007a147"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:15:06.936877 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:06.936779 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d250128-4d32-4bfa-bf99-f5566007a147-maas-api-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:15:07.682010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:07.681975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" event={"ID":"6d250128-4d32-4bfa-bf99-f5566007a147","Type":"ContainerDied","Data":"221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541"} Jul 13 20:15:07.682010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:07.681994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-641463-pcvjc" Jul 13 20:15:07.682225 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:07.682024 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221051c09265a7338ad1ea6692f321c81dbaa02e75f7cfd196c0e4eb396f7541" Jul 13 20:15:09.183563 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:09.183536 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:15:09.192975 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:09.192953 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:15:14.869852 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:14.869819 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:15:14.870329 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:14.870116 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" podUID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" containerName="maas-api" containerID="cri-o://027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67" gracePeriod=30 Jul 13 20:15:14.895358 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:14.895333 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc"] Jul 13 20:15:14.897941 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:14.897918 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-641463-29732895-wpckc"] Jul 13 20:15:14.960947 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:14.960919 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48443d4-367c-4213-984d-f9e618d2daff" path="/var/lib/kubelet/pods/b48443d4-367c-4213-984d-f9e618d2daff/volumes" Jul 13 20:15:15.118532 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.118504 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:15:15.208467 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.208351 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgsmt\" (UniqueName: \"kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt\") pod \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " Jul 13 20:15:15.208467 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.208399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls\") pod \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " Jul 13 20:15:15.208647 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.208477 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca\") pod \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\" (UID: \"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7\") " Jul 13 20:15:15.208895 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.208864 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" (UID: "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:15:15.210628 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.210604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" (UID: "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:15:15.210693 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.210676 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt" (OuterVolumeSpecName: "kube-api-access-vgsmt") pod "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" (UID: "1efc61dc-0ec6-4a2c-b51e-c750aad69bd7"). InnerVolumeSpecName "kube-api-access-vgsmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:15:15.309413 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.309383 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-openshift-service-ca\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:15:15.309413 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.309412 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsmt\" (UniqueName: \"kubernetes.io/projected/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-kube-api-access-vgsmt\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:15:15.309660 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.309469 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7-maas-api-tls\") on node \"ip-10-0-135-83.ec2.internal\" DevicePath \"\"" Jul 13 20:15:15.730534 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.730502 2582 generic.go:356] "Generic (PLEG): container finished" podID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" containerID="027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67" exitCode=0 Jul 13 20:15:15.730718 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.730545 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" Jul 13 20:15:15.730718 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.730594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" event={"ID":"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7","Type":"ContainerDied","Data":"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67"} Jul 13 20:15:15.730718 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.730667 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j" event={"ID":"1efc61dc-0ec6-4a2c-b51e-c750aad69bd7","Type":"ContainerDied","Data":"27608f91e9dcc729be85869b4c6048a1dbdab18de81fcc361607bd64529474cc"} Jul 13 20:15:15.730718 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.730689 2582 scope.go:117] "RemoveContainer" containerID="027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67" Jul 13 20:15:15.740292 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.740276 2582 scope.go:117] "RemoveContainer" containerID="027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67" Jul 13 20:15:15.740644 ip-10-0-135-83 kubenswrapper[2582]: E0713 20:15:15.740623 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67\": container with ID starting with 027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67 not found: ID does not exist" containerID="027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67" Jul 13 20:15:15.740728 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.740656 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67"} err="failed to get container status \"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67\": rpc error: code = NotFound desc = could not find container \"027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67\": container with ID starting with 027974c6fdc7673efea206d13b6354e57d8615c6a62903d0a7ea7ab8655c6b67 not found: ID does not exist" Jul 13 20:15:15.754660 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.754637 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:15:15.758241 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:15.758218 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-641463-76777f6bb7-7g84j"] Jul 13 20:15:16.961964 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:16.961929 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" path="/var/lib/kubelet/pods/1efc61dc-0ec6-4a2c-b51e-c750aad69bd7/volumes" Jul 13 20:15:35.890037 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:35.889999 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zhprd_de327392-716e-4a5f-b528-f04aa997e9c0/manager/0.log" Jul 13 20:15:36.008154 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:36.008120 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xwpxw_92ca9fff-de3e-4b11-a32e-0ab053813325/manager/0.log" Jul 13 20:15:36.260786 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:36.260702 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sc5j8_84c3e3bc-556c-4cc2-b5da-fa1402cbcca2/server/0.log" Jul 13 20:15:36.385702 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:36.385675 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f8rkz_e66b7d8e-f0fb-474b-b9c2-1d408f36abe5/manager/0.log" Jul 13 20:15:36.626246 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:36.626219 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699d4944b8-8pmxf_c02d48ba-0a78-4ebc-9d0a-dc4cf449d947/copy-manifests/0.log" Jul 13 20:15:36.638377 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:36.638348 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699d4944b8-8pmxf_c02d48ba-0a78-4ebc-9d0a-dc4cf449d947/manager/0.log" Jul 13 20:15:37.460641 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:37.460613 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732895-8986f_e9928cdb-f826-49e8-925a-5dd4c22db0d1/cleanup/0.log" Jul 13 20:15:37.565188 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:37.565160 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-29d8d858-2bj4g_9432562d-02f7-496a-8d2c-3e1f289e96eb/revoke-keys/0.log" Jul 13 20:15:37.672981 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:37.672951 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-facb3ea1-j22rw_3269aabc-cf9b-42ab-af4f-2cefc83dac25/revoke-keys/0.log" Jul 13 20:15:37.783402 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:37.783320 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-ebcb0c-48mgn_e59db086-2f35-4625-b6c0-69613a619f83/revoke-keys/0.log" Jul 13 20:15:37.888928 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:37.888904 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-3e52c5-zswjv_312df522-7e67-4387-adbb-5b3c51af9243/revoke-keys/0.log" Jul 13 20:15:38.122363 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.122337 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-416f95-f4gwk_9b7dd2c1-54ce-4351-ac62-5e41b52b4676/revoke-keys/0.log" Jul 13 20:15:38.230920 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.230895 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-6d9be2-gqh8t_ad237892-6781-4d28-ad9b-6cddda7f2bed/revoke-keys/0.log" Jul 13 20:15:38.450218 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.450128 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-02a26ce0-ll5lc_8eb1e7f8-46b5-4832-a611-68e3fc7ec802/revoke-keys/0.log" Jul 13 20:15:38.555328 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.555298 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-d94baa8e-8lgpf_4170062d-38eb-4127-bcd9-10213996be97/revoke-keys/0.log" Jul 13 20:15:38.667561 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.667529 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-939152-4bgbk_1106abd1-1755-4589-b624-970be427ec72/revoke-keys/0.log" Jul 13 20:15:38.775736 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:38.775666 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-641463-pcvjc_6d250128-4d32-4bfa-bf99-f5566007a147/revoke-keys/0.log" Jul 13 20:15:39.695647 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.695620 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/util/0.log" Jul 13 20:15:39.702650 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.702632 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/pull/0.log" Jul 13 20:15:39.711668 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.711639 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/extract/0.log" Jul 13 20:15:39.827556 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.827529 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/util/0.log" Jul 13 20:15:39.833763 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.833741 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/pull/0.log" Jul 13 20:15:39.840575 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.840559 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/extract/0.log" Jul 13 20:15:39.953986 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.953869 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/util/0.log" Jul 13 20:15:39.959560 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.959538 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/pull/0.log" Jul 13 20:15:39.965891 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:39.965874 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/extract/0.log" Jul 13 20:15:40.081861 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:40.081835 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/util/0.log" Jul 13 20:15:40.088513 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:40.088492 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/pull/0.log" Jul 13 20:15:40.095232 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:40.095212 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/extract/0.log" Jul 13 20:15:40.335287 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:40.335174 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pc5dd_e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61/manager/0.log" Jul 13 20:15:41.029125 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:41.029083 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hwwm4_58d510ca-ca82-473f-9e4d-2104460feca9/manager/0.log" Jul 13 20:15:41.515721 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:41.515694 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6lshj_51549055-9645-4fac-b63d-24323f766e19/discovery/0.log" Jul 13 20:15:41.626716 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:41.626675 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6996ff5f69-h7gq5_c39706d5-1853-4778-ace2-072e8e80bb0c/kube-auth-proxy/0.log" Jul 13 20:15:41.985994 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:41.985966 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-vdqx7_3ead7469-242a-4086-9e62-f2432fcf3633/payload-pre-processing/0.log" Jul 13 20:15:42.234444 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:42.234401 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5747877c9b-7jg6t_dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2/router/0.log" Jul 13 20:15:42.581987 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:42.581959 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xq7qb_d577c41d-482b-4188-b73d-2bb28157753f/main/0.log" Jul 13 20:15:42.958846 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:42.958815 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5lkn6_f0597483-a872-4ae4-a35d-f63b3d5d1cdd/main/0.log" Jul 13 20:15:50.447800 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:50.447768 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qb8q5_d918169a-694f-4831-b7f0-9844aa3acecd/global-pull-secret-syncer/0.log" Jul 13 20:15:50.572049 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:50.572010 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rpmz9_a1cc9f40-bab1-4714-83dd-bd8e75e89a92/konnectivity-agent/0.log" Jul 13 20:15:50.640162 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:50.640123 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-83.ec2.internal_5b69fff8245d16238da7871546a70633/haproxy/0.log" Jul 13 20:15:54.770631 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.770599 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/extract/0.log" Jul 13 20:15:54.796571 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.796546 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/util/0.log" Jul 13 20:15:54.828985 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.828953 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qg7zf_522bab15-3c3b-4c40-bcd7-2ace49a7c3d4/pull/0.log" Jul 13 20:15:54.910851 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.910827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/extract/0.log" Jul 13 20:15:54.972030 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.971938 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/util/0.log" Jul 13 20:15:54.997751 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:54.997714 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qwl4k_23d19676-1cae-45cc-8b7a-70abca3dd1d6/pull/0.log" Jul 13 20:15:55.022976 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.022956 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/extract/0.log" Jul 13 20:15:55.042792 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.042771 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/util/0.log" Jul 13 20:15:55.062437 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.062393 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hz52v_675d77cd-4578-4afa-a15a-abf9448ac330/pull/0.log" Jul 13 20:15:55.087730 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.087708 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/extract/0.log" Jul 13 20:15:55.110141 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.110116 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/util/0.log" Jul 13 20:15:55.131728 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.131710 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w9lv8_7433e734-b7e5-4112-9f00-ab13a6d97f94/pull/0.log" Jul 13 20:15:55.457883 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.457850 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pc5dd_e49b46c1-2fcb-4d9d-b7d1-4a59a3eaeb61/manager/0.log" Jul 13 20:15:55.767503 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:55.767398 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hwwm4_58d510ca-ca82-473f-9e4d-2104460feca9/manager/0.log" Jul 13 20:15:57.760585 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:57.760558 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-6m4h5_2b4e99e2-7ed5-4281-93c5-976864b6035d/cluster-monitoring-operator/0.log" Jul 13 20:15:57.874797 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:57.874776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54dbd485fd-7c4dc_ac9d5c01-b50e-4e05-9a5d-16b6fc276983/metrics-server/0.log" Jul 13 20:15:58.007273 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.007247 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxhd4_1ef930f6-17e0-4b8d-acf0-19be8d6a6579/node-exporter/0.log" Jul 13 20:15:58.030911 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.030827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxhd4_1ef930f6-17e0-4b8d-acf0-19be8d6a6579/kube-rbac-proxy/0.log" Jul 13 20:15:58.051597 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.051578 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jxhd4_1ef930f6-17e0-4b8d-acf0-19be8d6a6579/init-textfile/0.log" Jul 13 20:15:58.190611 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.190575 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t84w9_f25a3ddf-aec3-4829-88b0-15ea49cef014/kube-rbac-proxy-main/0.log" Jul 13 20:15:58.220909 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.220875 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t84w9_f25a3ddf-aec3-4829-88b0-15ea49cef014/kube-rbac-proxy-self/0.log" Jul 13 20:15:58.245463 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.245413 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t84w9_f25a3ddf-aec3-4829-88b0-15ea49cef014/openshift-state-metrics/0.log" Jul 13 20:15:58.578326 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.578301 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/thanos-query/0.log" Jul 13 20:15:58.597482 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.597461 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/kube-rbac-proxy-web/0.log" Jul 13 20:15:58.615162 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.615144 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/kube-rbac-proxy/0.log" Jul 13 20:15:58.640688 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.640662 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/prom-label-proxy/0.log" Jul 13 20:15:58.658955 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.658935 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/kube-rbac-proxy-rules/0.log" Jul 13 20:15:58.681603 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:58.681586 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c7476c988-f9rsq_851b076a-286c-4a39-902d-ec4c2f4b681b/kube-rbac-proxy-metrics/0.log" Jul 13 20:15:59.954572 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:15:59.954502 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-v6r8m_c959e1e6-d8a9-4cfc-8a10-f8aa63f27e63/networking-console-plugin/0.log" Jul 13 20:16:01.063041 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:01.063014 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b798ddc6c-jlkxl_ed605944-4582-4d0f-96cc-a073aa8d99ef/console/0.log" Jul 13 20:16:02.278021 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.277982 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c"] Jul 13 20:16:02.278457 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278392 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" containerName="maas-api" Jul 13 20:16:02.278457 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278404 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" containerName="maas-api" Jul 13 20:16:02.278457 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278419 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d250128-4d32-4bfa-bf99-f5566007a147" containerName="revoke-keys" Jul 13 20:16:02.278457 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278440 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d250128-4d32-4bfa-bf99-f5566007a147" containerName="revoke-keys" Jul 13 20:16:02.278610 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278506 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d250128-4d32-4bfa-bf99-f5566007a147" containerName="revoke-keys" Jul 13 20:16:02.278610 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.278516 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efc61dc-0ec6-4a2c-b51e-c750aad69bd7" containerName="maas-api" Jul 13 20:16:02.281759 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.281743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.284262 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.284244 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mmgjw\"/\"kube-root-ca.crt\"" Jul 13 20:16:02.284341 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.284248 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mmgjw\"/\"openshift-service-ca.crt\"" Jul 13 20:16:02.285029 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.285007 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mmgjw\"/\"default-dockercfg-tzzw6\"" Jul 13 20:16:02.291627 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.291604 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c"] Jul 13 20:16:02.328508 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.328480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-lib-modules\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.328673 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.328520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-proc\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.328673 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.328538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-podres\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.328673 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.328611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-sys\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.328793 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.328708 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kstwq\" (UniqueName: \"kubernetes.io/projected/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-kube-api-access-kstwq\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429595 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kstwq\" (UniqueName: \"kubernetes.io/projected/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-kube-api-access-kstwq\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429742 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-lib-modules\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429742 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-proc\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429742 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-podres\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429742 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-sys\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429877 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-lib-modules\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429877 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-proc\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429877 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-sys\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.429877 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.429823 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-podres\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.438049 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.438024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kstwq\" (UniqueName: \"kubernetes.io/projected/d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0-kube-api-access-kstwq\") pod \"perf-node-gather-daemonset-td48c\" (UID: \"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0\") " pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.592813 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.592712 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:02.735748 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.735725 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c"] Jul 13 20:16:02.736628 ip-10-0-135-83 kubenswrapper[2582]: W0713 20:16:02.736603 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0f6ac54_d2c7_4ec8_bc8d_e60aaedb9ab0.slice/crio-012c94bdb638014b33ea5710f39d2df16043c5acbc1e69b3896a08f8ed4c8138 WatchSource:0}: Error finding container 012c94bdb638014b33ea5710f39d2df16043c5acbc1e69b3896a08f8ed4c8138: Status 404 returned error can't find the container with id 012c94bdb638014b33ea5710f39d2df16043c5acbc1e69b3896a08f8ed4c8138 Jul 13 20:16:02.890725 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.890704 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8g4dk_30a696cb-16cb-4c68-8d82-03dc5c9f5fdb/dns/0.log" Jul 13 20:16:02.908449 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.908406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8g4dk_30a696cb-16cb-4c68-8d82-03dc5c9f5fdb/kube-rbac-proxy/0.log" Jul 13 20:16:02.993672 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:02.993652 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t762r_d6cbfa7b-786e-425b-82e5-10d3faaf9790/dns-node-resolver/0.log" Jul 13 20:16:03.013657 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:03.013621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" event={"ID":"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0","Type":"ContainerStarted","Data":"be094c8e5fd61c6de9462b7677e6bb0af74a1f40f74c00cd88f2de485db2ab27"} Jul 13 20:16:03.013803 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:03.013678 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" event={"ID":"d0f6ac54-d2c7-4ec8-bc8d-e60aaedb9ab0","Type":"ContainerStarted","Data":"012c94bdb638014b33ea5710f39d2df16043c5acbc1e69b3896a08f8ed4c8138"} Jul 13 20:16:03.013803 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:03.013710 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:03.031870 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:03.031831 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" podStartSLOduration=1.031817392 podStartE2EDuration="1.031817392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:16:03.028838635 +0000 UTC m=+3054.590236708" watchObservedRunningTime="2026-07-13 20:16:03.031817392 +0000 UTC m=+3054.593215435" Jul 13 20:16:03.612729 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:03.612699 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccvgs_33ea8156-02d6-4487-8148-d6c4c28fa8e0/node-ca/0.log" Jul 13 20:16:04.757606 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:04.757578 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6lshj_51549055-9645-4fac-b63d-24323f766e19/discovery/0.log" Jul 13 20:16:04.777411 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:04.777387 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6996ff5f69-h7gq5_c39706d5-1853-4778-ace2-072e8e80bb0c/kube-auth-proxy/0.log" Jul 13 20:16:04.882606 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:04.882577 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-vdqx7_3ead7469-242a-4086-9e62-f2432fcf3633/payload-pre-processing/0.log" Jul 13 20:16:04.934372 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:04.934347 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5747877c9b-7jg6t_dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2/router/0.log" Jul 13 20:16:05.612228 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:05.612199 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l4dlq_824435cf-d217-414d-aaa1-d54901a2dc69/serve-healthcheck-canary/0.log" Jul 13 20:16:06.102574 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:06.102553 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-g76j9_2b46d62f-d1a6-41a7-ab51-0064bb54a1bb/insights-operator/0.log" Jul 13 20:16:06.103036 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:06.102942 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-g76j9_2b46d62f-d1a6-41a7-ab51-0064bb54a1bb/insights-operator/1.log" Jul 13 20:16:06.121390 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:06.121371 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhvqg_e06a0954-27f7-4273-aa18-63542590dce6/kube-rbac-proxy/0.log" Jul 13 20:16:06.139689 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:06.139655 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhvqg_e06a0954-27f7-4273-aa18-63542590dce6/exporter/0.log" Jul 13 20:16:06.159349 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:06.159326 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhvqg_e06a0954-27f7-4273-aa18-63542590dce6/extractor/0.log" Jul 13 20:16:08.424028 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.423999 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zhprd_de327392-716e-4a5f-b528-f04aa997e9c0/manager/0.log" Jul 13 20:16:08.557765 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.557722 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xwpxw_92ca9fff-de3e-4b11-a32e-0ab053813325/manager/0.log" Jul 13 20:16:08.712342 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.712262 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sc5j8_84c3e3bc-556c-4cc2-b5da-fa1402cbcca2/server/0.log" Jul 13 20:16:08.773381 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.773341 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f8rkz_e66b7d8e-f0fb-474b-b9c2-1d408f36abe5/manager/0.log" Jul 13 20:16:08.884859 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.884831 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699d4944b8-8pmxf_c02d48ba-0a78-4ebc-9d0a-dc4cf449d947/manager/0.log" Jul 13 20:16:08.899520 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:08.899499 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699d4944b8-8pmxf_c02d48ba-0a78-4ebc-9d0a-dc4cf449d947/copy-manifests/0.log" Jul 13 20:16:09.031097 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:09.031016 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mmgjw/perf-node-gather-daemonset-td48c" Jul 13 20:16:10.998988 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:10.998953 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6lshj_51549055-9645-4fac-b63d-24323f766e19/discovery/0.log" Jul 13 20:16:11.011998 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:11.011968 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6996ff5f69-h7gq5_c39706d5-1853-4778-ace2-072e8e80bb0c/kube-auth-proxy/0.log" Jul 13 20:16:11.116010 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:11.115983 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-vdqx7_3ead7469-242a-4086-9e62-f2432fcf3633/payload-pre-processing/0.log" Jul 13 20:16:11.157222 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:11.157146 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5747877c9b-7jg6t_dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2/router/0.log" Jul 13 20:16:12.695199 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:12.695114 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6lshj_51549055-9645-4fac-b63d-24323f766e19/discovery/0.log" Jul 13 20:16:12.706086 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:12.706064 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6996ff5f69-h7gq5_c39706d5-1853-4778-ace2-072e8e80bb0c/kube-auth-proxy/0.log" Jul 13 20:16:12.788001 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:12.787969 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-vdqx7_3ead7469-242a-4086-9e62-f2432fcf3633/payload-pre-processing/0.log" Jul 13 20:16:12.822798 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:12.822771 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5747877c9b-7jg6t_dcb8a5e6-05d7-4953-ab1d-709e18f9b4b2/router/0.log" Jul 13 20:16:15.836259 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:15.836228 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-sqtk9_167e0ad5-cd81-4406-b8da-602edba14d4b/kube-storage-version-migrator-operator/1.log" Jul 13 20:16:15.837126 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:15.837105 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-sqtk9_167e0ad5-cd81-4406-b8da-602edba14d4b/kube-storage-version-migrator-operator/0.log" Jul 13 20:16:16.836176 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.836151 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/kube-multus-additional-cni-plugins/0.log" Jul 13 20:16:16.865256 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.865228 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/egress-router-binary-copy/0.log" Jul 13 20:16:16.893336 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.893312 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/cni-plugins/0.log" Jul 13 20:16:16.914819 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.914795 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/bond-cni-plugin/0.log" Jul 13 20:16:16.949558 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.949528 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/routeoverride-cni/0.log" Jul 13 20:16:16.969680 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.969652 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/whereabouts-cni-bincopy/0.log" Jul 13 20:16:16.991640 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:16.991617 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5dkxv_99b4cd6a-3faa-42c2-9334-a0ed6e7ef4bb/whereabouts-cni/0.log" Jul 13 20:16:17.553087 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:17.553057 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wrcgk_c07c93c3-afc5-44b6-a043-702914d1a37e/kube-multus/0.log" Jul 13 20:16:17.622209 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:17.622174 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qx6r_3a9254d4-7197-48a0-a784-bb398df08092/network-metrics-daemon/0.log" Jul 13 20:16:17.642370 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:17.642339 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qx6r_3a9254d4-7197-48a0-a784-bb398df08092/kube-rbac-proxy/0.log" Jul 13 20:16:18.940471 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:18.940445 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-controller/0.log" Jul 13 20:16:18.953727 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:18.953696 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/0.log" Jul 13 20:16:18.954162 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:18.954145 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovn-acl-logging/1.log" Jul 13 20:16:18.973547 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:18.973514 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/kube-rbac-proxy-node/0.log" Jul 13 20:16:18.992016 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:18.991990 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/kube-rbac-proxy-ovn-metrics/0.log" Jul 13 20:16:19.007569 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:19.007545 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/northd/0.log" Jul 13 20:16:19.024027 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:19.024007 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/nbdb/0.log" Jul 13 20:16:19.041484 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:19.041464 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/sbdb/0.log" Jul 13 20:16:19.091876 ip-10-0-135-83 kubenswrapper[2582]: I0713 20:16:19.091834 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q55l7_733e4d2b-0d31-46ae-ac4c-ea4b0d9763ea/ovnkube-controller/0.log"