Jul 13 14:52:16.974081 ip-10-0-134-241 systemd[1]: Starting Kubernetes Kubelet... Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:52:17.427767 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.424716 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430875 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430890 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430894 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430897 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430900 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430902 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430906 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430909 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430914 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430918 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430921 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430924 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430927 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430930 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430933 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430936 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430939 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430941 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.444064 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430944 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430947 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430950 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430952 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430955 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430958 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430961 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430963 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430966 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430974 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430976 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430979 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430982 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430984 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430987 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430989 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430992 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430995 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.430998 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431000 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.458628 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431003 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431006 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431008 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431011 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431013 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431017 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431019 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431023 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431026 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431028 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431031 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431033 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431036 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431038 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431041 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431043 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431046 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431048 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431050 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431053 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.460459 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431055 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431058 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431061 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431065 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431068 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431071 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431073 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431076 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431078 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431081 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431084 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431087 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431089 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431091 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431094 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431096 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431099 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431102 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431104 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431107 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.462882 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431109 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431112 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431115 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431117 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431120 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431123 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431126 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431128 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431131 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431133 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431136 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431138 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431140 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431143 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431146 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431149 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431152 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431156 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431160 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.470244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431165 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.431169 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432745 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432753 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432757 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432760 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432762 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432766 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432769 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432772 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432776 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432779 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432799 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432803 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432805 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432808 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432811 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432813 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432816 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432820 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.472466 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432822 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432825 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432827 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432830 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432832 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432835 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432838 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432842 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432846 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432849 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432852 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432858 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432861 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432863 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432867 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432869 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432872 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432874 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432877 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432880 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.474374 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432882 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432885 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432887 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432891 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432894 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432896 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432899 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432901 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432904 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432907 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432909 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432912 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432924 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432927 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432929 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432932 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432934 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432938 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432941 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.476738 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432945 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432949 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432952 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432954 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432959 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432961 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432964 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432967 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432969 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432971 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432975 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432978 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432981 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432984 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432986 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432988 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432991 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432993 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432996 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.490186 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.432998 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433001 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433003 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433006 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433009 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433011 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433014 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433017 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433019 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433022 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433024 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433028 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433030 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433033 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433037 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433040 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433042 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433046 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433048 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433051 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.492375 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433054 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433056 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433059 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433436 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433441 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433444 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433447 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433449 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433452 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433455 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433457 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433460 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433462 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433465 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433468 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433472 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433476 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433486 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433489 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433492 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.494269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433495 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433498 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433500 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433502 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433505 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433508 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433511 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433514 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433516 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433521 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433525 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433528 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433531 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433534 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433537 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433540 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433542 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433545 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433548 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433550 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.497056 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433553 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433555 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433558 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433560 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433562 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433565 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433567 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433570 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433573 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433575 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433577 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433580 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433583 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433585 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433588 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433590 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433593 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433595 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433599 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433601 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.579166 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433604 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.503840 ip-10-0-134-241 systemd[1]: Started Kubernetes Kubelet. Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433608 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433610 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433612 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433615 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433618 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433620 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433623 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433625 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433628 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433632 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433634 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433637 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433639 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433642 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433644 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433646 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433649 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433652 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433654 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.730934 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433657 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433660 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433662 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433665 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433667 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433670 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433673 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433675 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433678 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433680 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433684 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433686 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433689 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433692 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433695 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433698 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433700 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433704 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433708 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433710 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.737561 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433713 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.433715 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433809 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433830 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433839 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433844 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433848 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433851 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433856 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433860 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433863 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433866 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433869 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433873 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433876 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433879 2554 flags.go:64] FLAG: --cgroup-root="" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433881 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433884 2554 flags.go:64] FLAG: --client-ca-file="" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433888 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433891 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433896 2554 flags.go:64] FLAG: --cluster-domain="" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433899 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433904 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433907 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 14:52:17.738269 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433911 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433916 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433919 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433922 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433925 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433928 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433931 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433934 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433937 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433940 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433943 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433947 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433949 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433952 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433955 2554 flags.go:64] FLAG: --enable-server="true" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433958 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433962 2554 flags.go:64] FLAG: --event-burst="100" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433965 2554 flags.go:64] FLAG: --event-qps="50" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433968 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433971 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433974 2554 flags.go:64] FLAG: --eviction-hard="" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433977 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433980 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433984 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433987 2554 flags.go:64] FLAG: --eviction-soft="" Jul 13 14:52:17.738982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433989 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433992 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433996 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.433999 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434001 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434006 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434008 2554 flags.go:64] FLAG: --feature-gates="" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434012 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434016 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434019 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434022 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434025 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434028 2554 flags.go:64] FLAG: --help="false" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434031 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-134-241.ec2.internal" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434034 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434037 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434040 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434044 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434047 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434050 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434052 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434055 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434058 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 14:52:17.739701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434061 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434064 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434067 2554 flags.go:64] FLAG: --kube-reserved="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434070 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434072 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434076 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434078 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434081 2554 flags.go:64] FLAG: --lock-file="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434084 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434087 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434090 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434095 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434098 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434101 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434105 2554 flags.go:64] FLAG: --logging-format="text" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434108 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434111 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434115 2554 flags.go:64] FLAG: --manifest-url="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434118 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434122 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434125 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434129 2554 flags.go:64] FLAG: --max-pods="110" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434132 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434135 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434137 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 14:52:17.740454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434140 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434143 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434146 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434149 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434156 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434159 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434162 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434165 2554 flags.go:64] FLAG: --pod-cidr="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434168 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434173 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434176 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434179 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434182 2554 flags.go:64] FLAG: --port="10250" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434185 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434188 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f78fafad59b969ed" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434191 2554 flags.go:64] FLAG: --qos-reserved="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434194 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434197 2554 flags.go:64] FLAG: --register-node="true" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434199 2554 flags.go:64] FLAG: --register-with-taints="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434203 2554 flags.go:64] FLAG: --registry-burst="10" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434207 2554 flags.go:64] FLAG: --registry-qps="5" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434209 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434214 2554 flags.go:64] FLAG: --reserved-memory="" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434217 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434221 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 14:52:17.741233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434224 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434227 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434230 2554 flags.go:64] FLAG: --runonce="false" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434233 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434236 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434239 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434242 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434245 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434248 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434250 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434253 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434256 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434259 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434262 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434265 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434268 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434271 2554 flags.go:64] FLAG: --system-cgroups="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434274 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434279 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434282 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434285 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434289 2554 flags.go:64] FLAG: --tls-min-version="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434292 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434295 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434297 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 14:52:17.741989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434300 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434303 2554 flags.go:64] FLAG: --v="2" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434307 2554 flags.go:64] FLAG: --version="false" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434312 2554 flags.go:64] FLAG: --vmodule="" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434316 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.434319 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434435 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434439 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434441 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434444 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434448 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434450 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434453 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434456 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434459 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434462 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434464 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434468 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434471 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434473 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434476 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434478 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.742711 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434481 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434483 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434486 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434488 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434491 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434493 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434496 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434499 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434501 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434504 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434506 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434509 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434511 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434514 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434518 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434521 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434524 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434527 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434530 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434532 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.743372 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434535 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434537 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434541 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434543 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434546 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434550 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434553 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434556 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434559 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434561 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434563 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434566 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434569 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434571 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434573 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434576 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434579 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434581 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434584 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.744014 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434586 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434589 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434591 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434593 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434596 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434598 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434601 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434604 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434608 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434612 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434615 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434617 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434620 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434622 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434625 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434629 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434632 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434635 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434638 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.744552 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434641 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434643 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434646 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434649 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434651 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434654 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434657 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434659 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434662 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434664 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434667 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434670 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434674 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434677 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434679 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434682 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434684 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434687 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434689 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.745169 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434691 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434695 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434698 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434703 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434705 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.434708 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.435567 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.443126 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.443146 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.443170 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.443186 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443260 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443266 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443269 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443272 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.745701 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443275 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443282 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443285 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443290 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443298 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443301 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443307 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443310 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443312 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443315 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443319 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443322 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443324 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443327 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443332 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443335 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443338 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443345 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443347 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443350 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.746221 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443355 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443358 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443360 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443363 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443368 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443379 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443382 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443384 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443389 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443396 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443398 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443403 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443406 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443408 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443413 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443416 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443418 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443421 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443427 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443430 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.746845 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443432 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443437 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443440 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443442 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443444 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443449 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443452 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443454 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443457 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443462 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443465 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443470 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443472 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443475 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443477 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443480 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443482 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443485 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443496 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443499 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.747409 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443502 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443508 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443510 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443513 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443518 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443522 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443525 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443530 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443533 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443536 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443541 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443544 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443547 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443549 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443554 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443557 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443559 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443564 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443567 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443571 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.747936 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443576 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443579 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443581 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443584 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443588 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443591 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443593 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443596 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443600 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443602 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443605 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443616 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443620 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443625 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.748503 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443628 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.443635 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443815 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443822 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443826 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443830 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443833 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443836 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443839 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443844 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443846 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443849 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443854 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443857 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443860 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:52:17.748898 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443865 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443868 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443872 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443876 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443878 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443881 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443885 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443887 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443890 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443892 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443895 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443898 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443900 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443903 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443907 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443918 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443921 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443924 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443929 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443932 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:52:17.749395 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443934 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443937 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443941 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443943 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443946 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443950 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443952 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443955 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443958 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443960 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443963 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443965 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443968 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443971 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443977 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443980 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443983 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443985 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443988 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:52:17.860290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443991 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443994 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443997 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.443999 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444002 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444004 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444007 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444009 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444012 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444020 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444023 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444026 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444028 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444031 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444034 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444036 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444039 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444041 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444044 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444046 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:52:17.861111 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444049 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444051 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444054 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444057 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444059 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444062 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444064 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444068 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444071 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444074 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444077 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444079 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444082 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444084 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444087 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444089 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444093 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444096 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444099 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444102 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:52:17.862440 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444105 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444107 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444116 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444119 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444122 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444124 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:17.444128 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.444133 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.444255 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.447751 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.449301 2554 server.go:1019] "Starting client certificate rotation" Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.449424 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.449455 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:52:17.863041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.452268 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.460921 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.473409 2554 log.go:25] "Validated CRI v1 runtime API" Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.475873 2554 log.go:25] "Validated CRI v1 image API" Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.476737 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.480031 2554 fs.go:135] Filesystem UUIDs: map[50d4d88d-ab59-4c8c-b894-23de15177011:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eb6843d7-8954-4f67-b61c-ab5a9c9e9218:/dev/nvme0n1p4] Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.480045 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 14:52:17.863405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.484385 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.484284 2554 manager.go:216] Machine: {Timestamp:2026-07-13 14:52:17.48378216 +0000 UTC m=+0.374457253 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100202 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297560664cc9c5c3fead40ad7f72bc SystemUUID:ec297560-664c-c9c5-c3fe-ad40ad7f72bc BootID:a23e167c-e4b3-4153-a2f1-f6ec36c9305e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f8:db:9d:a4:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f8:db:9d:a4:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:f7:30:7e:69:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.484438 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.484528 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.485767 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.485804 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.486677 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.486687 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.486694 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.486707 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.487851 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492113 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492245 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492278 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492291 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492314 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492326 2554 kubelet.go:398] "Adding apiserver pod source" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492336 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 14:52:17.863701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492757 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.492772 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.495333 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.497208 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.497235 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498043 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498063 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498073 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498083 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498096 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498104 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498116 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498135 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498146 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498156 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498168 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.498181 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.499088 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.499098 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.502889 2554 server.go:1268] "Started kubelet" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.503430 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.504341 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.504363 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.504434 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.504562 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.504572 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.505050 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.506035 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.510673 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.510958 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-241.ec2.internal.18c1e104a65d2867 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-241.ec2.internal,UID:ip-10-0-134-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-241.ec2.internal,},FirstTimestamp:2026-07-13 14:52:17.502668903 +0000 UTC m=+0.393344002,LastTimestamp:2026-07-13 14:52:17.502668903 +0000 UTC m=+0.393344002,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-241.ec2.internal,}" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.512881 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.512895 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.512979 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513589 2554 factory.go:55] Registering systemd factory Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513603 2554 factory.go:223] Registration of the systemd container factory successfully Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513620 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513621 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513644 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513747 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513755 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513840 2554 factory.go:153] Registering CRI-O factory Jul 13 14:52:18.019567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513854 2554 factory.go:223] Registration of the crio container factory successfully Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513899 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513922 2554 factory.go:103] Registering Raw factory Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.513939 2554 manager.go:1196] Started watching for new ooms in manager Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.514239 2554 manager.go:349] Starting recovery of all containers Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.520857 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.523166 2554 manager.go:354] Recovery completed Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.523952 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.524009 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.527469 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.529515 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.529551 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.529568 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.530113 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.530113 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.530201 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.530219 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.531284 2554 policy_none.go:49] "None policy: Start" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.531295 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.531304 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.531921 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-241.ec2.internal.18c1e104a7f70c1f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-241.ec2.internal,UID:ip-10-0-134-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-241.ec2.internal,},FirstTimestamp:2026-07-13 14:52:17.529531423 +0000 UTC m=+0.420206521,LastTimestamp:2026-07-13 14:52:17.529531423 +0000 UTC m=+0.420206521,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-241.ec2.internal,}" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.532252 2554 policy_none.go:47] "Start" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.532358 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.532384 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.532398 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.541421 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r4r6z" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.542074 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-241.ec2.internal.18c1e104a7f7791d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-241.ec2.internal,UID:ip-10-0-134-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-241.ec2.internal,},FirstTimestamp:2026-07-13 14:52:17.529559325 +0000 UTC m=+0.420234418,LastTimestamp:2026-07-13 14:52:17.529559325 +0000 UTC m=+0.420234418,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-241.ec2.internal,}" Jul 13 14:52:18.020697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.549544 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r4r6z" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.552715 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-241.ec2.internal.18c1e104a7f7b263 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-241.ec2.internal,UID:ip-10-0-134-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-241.ec2.internal,},FirstTimestamp:2026-07-13 14:52:17.529573987 +0000 UTC m=+0.420249082,LastTimestamp:2026-07-13 14:52:17.529573987 +0000 UTC m=+0.420249082,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-241.ec2.internal,}" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.597801 2554 manager.go:337] "Starting Device Plugin manager" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.597828 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.597838 2554 server.go:87] "Starting device plugin registration server" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.598051 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.598060 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.598153 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.598375 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.598384 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.598635 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.598691 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.672158 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.672203 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.672225 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.672260 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.674859 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.698988 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.700163 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.700193 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.700207 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.700232 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.710181 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.710200 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-241.ec2.internal\": node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.735719 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.773143 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal"] Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.773209 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.776305 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.776336 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.776347 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.777548 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.777660 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.777690 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778261 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778286 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778296 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778262 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.021506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778360 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.778374 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.779682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.779713 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.780392 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.780419 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.780436 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.806042 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-241.ec2.internal\" not found" node="ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.811456 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-241.ec2.internal\" not found" node="ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.814211 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98260bfd566f14ae3c27b8cadea6e987-config\") pod \"kube-apiserver-proxy-ip-10-0-134-241.ec2.internal\" (UID: \"98260bfd566f14ae3c27b8cadea6e987\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.814231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.814249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.836228 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.914889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98260bfd566f14ae3c27b8cadea6e987-config\") pod \"kube-apiserver-proxy-ip-10-0-134-241.ec2.internal\" (UID: \"98260bfd566f14ae3c27b8cadea6e987\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.914918 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.914935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.914975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.914991 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98260bfd566f14ae3c27b8cadea6e987-config\") pod \"kube-apiserver-proxy-ip-10-0-134-241.ec2.internal\" (UID: \"98260bfd566f14ae3c27b8cadea6e987\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:17.915000 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a4bac0d5aef81006c66c4c79f72062e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal\" (UID: \"a4bac0d5aef81006c66c4c79f72062e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.022515 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:17.937005 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.189929 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.037743 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.189929 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.108946 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.189929 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.114066 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:18.189929 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.138590 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.317728 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.239046 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.416931 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.339604 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.416931 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.380651 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:18.522195 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.440366 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.522195 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.448541 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 14:52:18.522195 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.448672 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 14:52:18.522195 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.448679 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 14:52:18.643289 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.541121 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.643289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.554314 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 14:47:18 +0000 UTC" deadline="2028-02-14 09:41:46.364469468 +0000 UTC" Jul 13 14:52:18.643289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.554351 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13938h49m27.810121772s" Jul 13 14:52:18.643289 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.641196 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.758944 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:18.739001 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:52:18.758944 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.741628 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.874149 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.842158 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:18.994032 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:18.942973 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:19.113837 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.044063 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.144905 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-241.ec2.internal\" not found" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.211690 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.211731 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.211751 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.214174 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:19.246056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.219690 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:19.410671 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.313941 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" Jul 13 14:52:19.410671 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.327135 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:52:19.410671 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.328380 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" Jul 13 14:52:19.410671 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.345667 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.495939 2554 apiserver.go:52] "Watching apiserver" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.503485 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.505713 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-g2q2r","openshift-network-operator/iptables-alerter-xmz8z","kube-system/konnectivity-agent-4wf4g","kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal","openshift-dns/node-resolver-m2k8m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-m7rh6","openshift-multus/multus-mppss","openshift-ovn-kubernetes/ovnkube-node-khk5x","kube-system/global-pull-secret-syncer-6mfkf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8","openshift-cluster-node-tuning-operator/tuned-x2q8r","openshift-image-registry/node-ca-bvxdz","openshift-multus/network-metrics-daemon-vrlkt"] Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.513583 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524600 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524624 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-sys-fs\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-socket-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524657 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-device-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524712 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.524743 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-registration-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.529881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.525819 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.566962 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hm7kv" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.573703 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hm7kv" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625302 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-sys-fs\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625341 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-socket-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-device-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625423 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625448 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-registration-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625452 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-device-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625423 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-sys-fs\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625473 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625500 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-socket-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625558 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-registration-dir\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.625597 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.632691 2554 projected.go:291] Couldn't get configMap openshift-cluster-csi-drivers/kube-root-ca.crt: object "openshift-cluster-csi-drivers"/"kube-root-ca.crt" not registered Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.632708 2554 projected.go:291] Couldn't get configMap openshift-cluster-csi-drivers/openshift-service-ca.crt: object "openshift-cluster-csi-drivers"/"openshift-service-ca.crt" not registered Jul 13 14:52:19.637062 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.632717 2554 projected.go:196] Error preparing data for projected volume kube-api-access-xcw28 for pod openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8: [object "openshift-cluster-csi-drivers"/"kube-root-ca.crt" not registered, object "openshift-cluster-csi-drivers"/"openshift-service-ca.crt" not registered] Jul 13 14:52:19.739184 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.632774 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28 podName:4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:20.132758578 +0000 UTC m=+3.023433676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xcw28" (UniqueName: "kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28") pod "aws-ebs-csi-driver-node-mgcq8" (UID: "4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b") : [object "openshift-cluster-csi-drivers"/"kube-root-ca.crt" not registered, object "openshift-cluster-csi-drivers"/"openshift-service-ca.crt" not registered] Jul 13 14:52:19.791127 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.791060 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:19.792685 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.792661 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.794459 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.794440 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 14:52:19.794619 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.794592 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.794715 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.794637 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.794715 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.794692 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jwlfp\"" Jul 13 14:52:19.795210 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.795189 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.795364 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.795341 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.795538 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.795346 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 14:52:19.795756 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.795728 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.795874 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.795728 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sxjpg\"" Jul 13 14:52:19.796626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.796605 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.796723 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.796690 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:19.797605 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.797585 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 14:52:19.798049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.798030 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.798148 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.798075 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 14:52:19.798439 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.798425 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tc4l2\"" Jul 13 14:52:19.799475 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.799450 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.800937 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.800846 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.801077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.801061 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.801246 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.801179 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zfbr\"" Jul 13 14:52:19.802002 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.801758 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.803122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.802965 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb2kk\"" Jul 13 14:52:19.803122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.802965 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.803274 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.803241 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 14:52:19.803865 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.803822 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.804412 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hj5cd\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.804472 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.804569 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.804750 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.804765 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.805059 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 14:52:19.805193 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.805093 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mppss" Jul 13 14:52:19.806652 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.806635 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.807363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.807347 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 14:52:19.807626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.807612 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gq9qr\"" Jul 13 14:52:19.809024 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809006 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.809117 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809095 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 14:52:19.809354 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809336 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 14:52:19.809435 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809390 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.809727 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809593 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.809727 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.809632 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 14:52:19.810594 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.810578 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.810721 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.810637 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:19.810721 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.810688 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4cfp4\"" Jul 13 14:52:19.810721 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.810704 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 14:52:19.811503 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.811487 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:52:19.811585 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.811543 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 14:52:19.811645 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.811609 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tr8cz\"" Jul 13 14:52:19.812047 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.812033 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:19.812132 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.812097 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:19.814721 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.814706 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 14:52:19.827579 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827558 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-node-log\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827676 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e9f752-fd9a-4f05-8c86-fb30760212c3-host\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.827676 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-etc-kubernetes\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.827676 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827620 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-env-overrides\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827676 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827640 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-modprobe-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.827824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827666 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc2e3881-4d9f-478e-afca-e481e662939b-host-slash\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.827824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-system-cni-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.827824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827722 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-multus-certs\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.827824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827739 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-systemd-units\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvpl\" (UniqueName: \"kubernetes.io/projected/a425f617-f69e-48d4-9995-52d948d5c86e-kube-api-access-tpvpl\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827850 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-sys\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.827984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827878 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-etc-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827909 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-bin\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.827984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-kubelet-config\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.827984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.827977 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02703252-d08b-45ff-9716-08cbe91b83e1-konnectivity-ca\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828033 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828064 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-kubelet\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828081 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-systemd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828112 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-tuned\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e9f752-fd9a-4f05-8c86-fb30760212c3-serviceca\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828146 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cnibin\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.828165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828160 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-cnibin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828182 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-socket-dir-parent\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-slash\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828251 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-tmp\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828276 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2tn\" (UniqueName: \"kubernetes.io/projected/56e9f752-fd9a-4f05-8c86-fb30760212c3-kube-api-access-8s2tn\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828302 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-k8s-cni-cncf-io\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828317 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-run\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828339 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-host\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828362 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnn6\" (UniqueName: \"kubernetes.io/projected/bc2e3881-4d9f-478e-afca-e481e662939b-kube-api-access-9rnn6\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-os-release\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.828398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828394 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-netns\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828421 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828439 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-dbus\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828456 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828471 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828499 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-multus-daemon-config\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-log-socket\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-netd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02703252-d08b-45ff-9716-08cbe91b83e1-agent-certs\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828569 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffnx\" (UniqueName: \"kubernetes.io/projected/72b450b5-9487-47a9-a267-e47ba75caa74-kube-api-access-2ffnx\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828600 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-script-lib\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828620 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828636 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-lib-modules\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-var-lib-kubelet\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828684 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dcs\" (UniqueName: \"kubernetes.io/projected/c601ac53-21fc-4809-80e6-b8cdc296e7e7-kube-api-access-b7dcs\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828706 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.828737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828731 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ps2\" (UniqueName: \"kubernetes.io/projected/213edeac-f8f8-4ffc-864c-829a72c409f3-kube-api-access-86ps2\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828757 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-cni-binary-copy\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828777 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-netns\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828796 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-bin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828830 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-config\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828846 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828862 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828885 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-os-release\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828921 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-hostroot\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-var-lib-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828973 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-kubernetes\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.828994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-conf\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829026 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc2e3881-4d9f-478e-afca-e481e662939b-iptables-alerter-script\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c601ac53-21fc-4809-80e6-b8cdc296e7e7-hosts-file\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829109 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c601ac53-21fc-4809-80e6-b8cdc296e7e7-tmp-dir\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829150 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-kubelet\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829179 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-ovn\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829206 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-system-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-multus\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829295 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829325 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829353 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysconfig\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhfc\" (UniqueName: \"kubernetes.io/projected/710e4f30-a65b-421f-a1fc-d738f21068ef-kube-api-access-kmhfc\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829416 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a425f617-f69e-48d4-9995-52d948d5c86e-ovn-node-metrics-cert\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829448 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-systemd\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829467 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qrr\" (UniqueName: \"kubernetes.io/projected/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-kube-api-access-l6qrr\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.829774 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.829483 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-conf-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.928290 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:19.928260 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98260bfd566f14ae3c27b8cadea6e987.slice/crio-4e25ef07270b9b06dd043a261fb1236f484bbaccea055ba182dfb0006257c9fa WatchSource:0}: Error finding container 4e25ef07270b9b06dd043a261fb1236f484bbaccea055ba182dfb0006257c9fa: Status 404 returned error can't find the container with id 4e25ef07270b9b06dd043a261fb1236f484bbaccea055ba182dfb0006257c9fa Jul 13 14:52:19.928591 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:19.928569 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bac0d5aef81006c66c4c79f72062e4.slice/crio-244d304d88299cad37c1c90813d0868834120b86020b947c22046f84a7d12197 WatchSource:0}: Error finding container 244d304d88299cad37c1c90813d0868834120b86020b947c22046f84a7d12197: Status 404 returned error can't find the container with id 244d304d88299cad37c1c90813d0868834120b86020b947c22046f84a7d12197 Jul 13 14:52:19.929993 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.929972 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02703252-d08b-45ff-9716-08cbe91b83e1-agent-certs\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.930096 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930001 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffnx\" (UniqueName: \"kubernetes.io/projected/72b450b5-9487-47a9-a267-e47ba75caa74-kube-api-access-2ffnx\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930096 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930018 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-script-lib\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.930096 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930043 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.930096 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930076 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-lib-modules\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930144 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-var-lib-kubelet\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930187 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dcs\" (UniqueName: \"kubernetes.io/projected/c601ac53-21fc-4809-80e6-b8cdc296e7e7-kube-api-access-b7dcs\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930213 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ps2\" (UniqueName: \"kubernetes.io/projected/213edeac-f8f8-4ffc-864c-829a72c409f3-kube-api-access-86ps2\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930261 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-cni-binary-copy\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.930315 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930284 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-var-lib-kubelet\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930332 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-netns\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930286 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-netns\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-bin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930391 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-config\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930442 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930464 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-os-release\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930487 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-hostroot\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930511 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-var-lib-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930547 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930558 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-kubernetes\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930586 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-conf\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930694 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-lib-modules\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930700 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-conf\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930730 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc2e3881-4d9f-478e-afca-e481e662939b-iptables-alerter-script\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.930768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930757 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c601ac53-21fc-4809-80e6-b8cdc296e7e7-hosts-file\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930786 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c601ac53-21fc-4809-80e6-b8cdc296e7e7-tmp-dir\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-kubelet\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-ovn\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930910 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-ovn\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930923 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c601ac53-21fc-4809-80e6-b8cdc296e7e7-hosts-file\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.930967 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931001 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-system-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931030 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-multus\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931056 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931083 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-cni-binary-copy\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931111 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysconfig\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhfc\" (UniqueName: \"kubernetes.io/projected/710e4f30-a65b-421f-a1fc-d738f21068ef-kube-api-access-kmhfc\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931180 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a425f617-f69e-48d4-9995-52d948d5c86e-ovn-node-metrics-cert\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931209 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-systemd\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931217 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c601ac53-21fc-4809-80e6-b8cdc296e7e7-tmp-dir\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.931635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931238 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qrr\" (UniqueName: \"kubernetes.io/projected/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-kube-api-access-l6qrr\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-conf-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-config\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-node-log\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e9f752-fd9a-4f05-8c86-fb30760212c3-host\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931335 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-hostroot\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931338 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-kubelet\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-etc-kubernetes\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931380 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-env-overrides\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931400 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-systemd\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931405 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-kubernetes\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931416 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-modprobe-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc2e3881-4d9f-478e-afca-e481e662939b-host-slash\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931460 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e9f752-fd9a-4f05-8c86-fb30760212c3-host\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931481 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc2e3881-4d9f-478e-afca-e481e662939b-host-slash\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-multus\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931539 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysconfig\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.932524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931556 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-system-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931581 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-ovn-kubernetes\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931596 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-cni-bin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931646 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931719 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-os-release\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931744 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931764 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-etc-kubernetes\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931405 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-conf-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931803 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-node-log\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931845 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-cni-dir\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-modprobe-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931924 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-system-cni-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-var-lib-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932024 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.931480 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-system-cni-dir\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932231 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-env-overrides\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932234 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-multus-certs\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932286 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-systemd-units\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.933427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932291 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-multus-certs\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932339 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-systemd-units\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvpl\" (UniqueName: \"kubernetes.io/projected/a425f617-f69e-48d4-9995-52d948d5c86e-kube-api-access-tpvpl\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932396 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-sys\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932418 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-etc-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932448 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a425f617-f69e-48d4-9995-52d948d5c86e-ovnkube-script-lib\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932467 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-bin\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932494 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-kubelet-config\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932527 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-sys\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932538 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-bin\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932593 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02703252-d08b-45ff-9716-08cbe91b83e1-konnectivity-ca\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932622 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932627 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc2e3881-4d9f-478e-afca-e481e662939b-iptables-alerter-script\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-etc-openvswitch\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-kubelet\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-kubelet-config\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.934382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-systemd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932701 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-var-lib-kubelet\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.932733 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932744 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-run-systemd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932761 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-tuned\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932789 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e9f752-fd9a-4f05-8c86-fb30760212c3-serviceca\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.932812 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:20.432788803 +0000 UTC m=+3.323463907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932862 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cnibin\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-cnibin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932915 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-socket-dir-parent\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-slash\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932959 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-tmp\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.932980 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2tn\" (UniqueName: \"kubernetes.io/projected/56e9f752-fd9a-4f05-8c86-fb30760212c3-kube-api-access-8s2tn\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933001 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-k8s-cni-cncf-io\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933024 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-run\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-host\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933091 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnn6\" (UniqueName: \"kubernetes.io/projected/bc2e3881-4d9f-478e-afca-e481e662939b-kube-api-access-9rnn6\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.935460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933117 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-os-release\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933143 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-netns\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933170 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933194 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-dbus\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933222 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933304 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-multus-daemon-config\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933337 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-log-socket\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933364 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-netd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933452 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-cni-netd\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933464 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02703252-d08b-45ff-9716-08cbe91b83e1-konnectivity-ca\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933453 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-host\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933515 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-cnibin\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933592 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-cnibin\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933651 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-multus-socket-dir-parent\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933701 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-slash\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933714 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-os-release\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933764 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-host-run-netns\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936237 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.933834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-run\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72b450b5-9487-47a9-a267-e47ba75caa74-host-run-k8s-cni-cncf-io\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934023 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b09ff867-1f8f-40fc-883f-7afd37b3b88b-dbus\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934079 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a425f617-f69e-48d4-9995-52d948d5c86e-log-socket\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934140 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-sysctl-d\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72b450b5-9487-47a9-a267-e47ba75caa74-multus-daemon-config\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934609 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e9f752-fd9a-4f05-8c86-fb30760212c3-serviceca\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934685 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02703252-d08b-45ff-9716-08cbe91b83e1-agent-certs\") pod \"konnectivity-agent-4wf4g\" (UID: \"02703252-d08b-45ff-9716-08cbe91b83e1\") " pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.934762 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.934891 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.934939 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:20.434923213 +0000 UTC m=+3.325598301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.935179 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.935681 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a425f617-f69e-48d4-9995-52d948d5c86e-ovn-node-metrics-cert\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.936085 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-etc-tuned\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.936877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.936213 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/710e4f30-a65b-421f-a1fc-d738f21068ef-tmp\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.939732 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.939711 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffnx\" (UniqueName: \"kubernetes.io/projected/72b450b5-9487-47a9-a267-e47ba75caa74-kube-api-access-2ffnx\") pod \"multus-mppss\" (UID: \"72b450b5-9487-47a9-a267-e47ba75caa74\") " pod="openshift-multus/multus-mppss" Jul 13 14:52:19.939958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.939941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ps2\" (UniqueName: \"kubernetes.io/projected/213edeac-f8f8-4ffc-864c-829a72c409f3-kube-api-access-86ps2\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:19.940102 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.940085 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dcs\" (UniqueName: \"kubernetes.io/projected/c601ac53-21fc-4809-80e6-b8cdc296e7e7-kube-api-access-b7dcs\") pod \"node-resolver-m2k8m\" (UID: \"c601ac53-21fc-4809-80e6-b8cdc296e7e7\") " pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:19.941077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.941053 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhfc\" (UniqueName: \"kubernetes.io/projected/710e4f30-a65b-421f-a1fc-d738f21068ef-kube-api-access-kmhfc\") pod \"tuned-x2q8r\" (UID: \"710e4f30-a65b-421f-a1fc-d738f21068ef\") " pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:19.942736 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.942715 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:19.942838 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.942740 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:19.942838 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.942752 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:19.942838 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:19.942791 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:20.442780172 +0000 UTC m=+3.333455259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:19.943054 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.942856 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvpl\" (UniqueName: \"kubernetes.io/projected/a425f617-f69e-48d4-9995-52d948d5c86e-kube-api-access-tpvpl\") pod \"ovnkube-node-khk5x\" (UID: \"a425f617-f69e-48d4-9995-52d948d5c86e\") " pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:19.943851 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.943810 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qrr\" (UniqueName: \"kubernetes.io/projected/160c96bd-e5f1-4bc8-9788-14dee3ac8af6-kube-api-access-l6qrr\") pod \"multus-additional-cni-plugins-m7rh6\" (UID: \"160c96bd-e5f1-4bc8-9788-14dee3ac8af6\") " pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:19.944819 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.944802 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnn6\" (UniqueName: \"kubernetes.io/projected/bc2e3881-4d9f-478e-afca-e481e662939b-kube-api-access-9rnn6\") pod \"iptables-alerter-xmz8z\" (UID: \"bc2e3881-4d9f-478e-afca-e481e662939b\") " pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:19.945130 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.945114 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2tn\" (UniqueName: \"kubernetes.io/projected/56e9f752-fd9a-4f05-8c86-fb30760212c3-kube-api-access-8s2tn\") pod \"node-ca-bvxdz\" (UID: \"56e9f752-fd9a-4f05-8c86-fb30760212c3\") " pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:19.970455 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:19.970436 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:20.122069 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.121965 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmz8z" Jul 13 14:52:20.131665 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.131635 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2e3881_4d9f_478e_afca_e481e662939b.slice/crio-991e483031716d65a3f4144c1323f520c5a4dba95ac907f1c17261ef674fc04a WatchSource:0}: Error finding container 991e483031716d65a3f4144c1323f520c5a4dba95ac907f1c17261ef674fc04a: Status 404 returned error can't find the container with id 991e483031716d65a3f4144c1323f520c5a4dba95ac907f1c17261ef674fc04a Jul 13 14:52:20.132884 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.132867 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:20.134916 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.134727 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:20.137266 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.137245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b-kube-api-access-xcw28\") pod \"aws-ebs-csi-driver-node-mgcq8\" (UID: \"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:20.140010 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.139982 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m2k8m" Jul 13 14:52:20.140841 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.140820 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02703252_d08b_45ff_9716_08cbe91b83e1.slice/crio-44a882b3b9b252c86a787ca32c1666e4e61dd0a8dbc40bce77a280e20160612b WatchSource:0}: Error finding container 44a882b3b9b252c86a787ca32c1666e4e61dd0a8dbc40bce77a280e20160612b: Status 404 returned error can't find the container with id 44a882b3b9b252c86a787ca32c1666e4e61dd0a8dbc40bce77a280e20160612b Jul 13 14:52:20.148539 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.148504 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc601ac53_21fc_4809_80e6_b8cdc296e7e7.slice/crio-35eaa1ff3a8fea88ec997d7c2eac35424e940a59cd3dec47f912c3ee9cb35274 WatchSource:0}: Error finding container 35eaa1ff3a8fea88ec997d7c2eac35424e940a59cd3dec47f912c3ee9cb35274: Status 404 returned error can't find the container with id 35eaa1ff3a8fea88ec997d7c2eac35424e940a59cd3dec47f912c3ee9cb35274 Jul 13 14:52:20.157511 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.157494 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bvxdz" Jul 13 14:52:20.164542 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.164513 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" Jul 13 14:52:20.164834 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.164810 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e9f752_fd9a_4f05_8c86_fb30760212c3.slice/crio-05c5a68ae08b98b0229a0aba58aa657a5877ab01ea6145ba2323842bf6b6d700 WatchSource:0}: Error finding container 05c5a68ae08b98b0229a0aba58aa657a5877ab01ea6145ba2323842bf6b6d700: Status 404 returned error can't find the container with id 05c5a68ae08b98b0229a0aba58aa657a5877ab01ea6145ba2323842bf6b6d700 Jul 13 14:52:20.172013 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.171990 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c96bd_e5f1_4bc8_9788_14dee3ac8af6.slice/crio-846d12bee5c6b22d9f6ad80d7f29af8a32690f91b379edb7224eb4f4c9768a62 WatchSource:0}: Error finding container 846d12bee5c6b22d9f6ad80d7f29af8a32690f91b379edb7224eb4f4c9768a62: Status 404 returned error can't find the container with id 846d12bee5c6b22d9f6ad80d7f29af8a32690f91b379edb7224eb4f4c9768a62 Jul 13 14:52:20.186266 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.186249 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mppss" Jul 13 14:52:20.193012 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.192990 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b450b5_9487_47a9_a267_e47ba75caa74.slice/crio-72fb7853d60c9ee5ad75350dd06f5fbbc51faf5558186b462a85ab5e0860f5d0 WatchSource:0}: Error finding container 72fb7853d60c9ee5ad75350dd06f5fbbc51faf5558186b462a85ab5e0860f5d0: Status 404 returned error can't find the container with id 72fb7853d60c9ee5ad75350dd06f5fbbc51faf5558186b462a85ab5e0860f5d0 Jul 13 14:52:20.207009 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.206990 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:20.213789 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.213772 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" Jul 13 14:52:20.214133 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.214108 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda425f617_f69e_48d4_9995_52d948d5c86e.slice/crio-1a8a7be68f3425e03c2c8ea4edd793215541fa0b77210eab52f54a7ff0a671a3 WatchSource:0}: Error finding container 1a8a7be68f3425e03c2c8ea4edd793215541fa0b77210eab52f54a7ff0a671a3: Status 404 returned error can't find the container with id 1a8a7be68f3425e03c2c8ea4edd793215541fa0b77210eab52f54a7ff0a671a3 Jul 13 14:52:20.221100 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.221080 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710e4f30_a65b_421f_a1fc_d738f21068ef.slice/crio-7dcef1fd4f047d66874535bea7f052225f9ffc73f07c4eea16c0f4bc81807ff1 WatchSource:0}: Error finding container 7dcef1fd4f047d66874535bea7f052225f9ffc73f07c4eea16c0f4bc81807ff1: Status 404 returned error can't find the container with id 7dcef1fd4f047d66874535bea7f052225f9ffc73f07c4eea16c0f4bc81807ff1 Jul 13 14:52:20.409819 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.409733 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" Jul 13 14:52:20.417663 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:20.417635 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb516a0_9aca_42f4_b3b0_cdb1f0b54e1b.slice/crio-31f6a71425b5a3999510b203d8e241cb9beabebd02cf5ac964afe9cf6dc70642 WatchSource:0}: Error finding container 31f6a71425b5a3999510b203d8e241cb9beabebd02cf5ac964afe9cf6dc70642: Status 404 returned error can't find the container with id 31f6a71425b5a3999510b203d8e241cb9beabebd02cf5ac964afe9cf6dc70642 Jul 13 14:52:20.436542 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.436516 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:20.436640 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.436563 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:20.436684 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.436658 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:20.436684 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.436660 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:20.436747 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.436705 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:21.436691036 +0000 UTC m=+4.327366118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:20.436747 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.436717 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:21.436711578 +0000 UTC m=+4.327386659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:20.537403 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.537358 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:20.538191 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.537606 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:20.538191 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.537631 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:20.538191 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.537645 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:20.538191 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:20.537708 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:21.537688767 +0000 UTC m=+4.428363849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:20.682631 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.682445 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mppss" event={"ID":"72b450b5-9487-47a9-a267-e47ba75caa74","Type":"ContainerStarted","Data":"72fb7853d60c9ee5ad75350dd06f5fbbc51faf5558186b462a85ab5e0860f5d0"} Jul 13 14:52:20.684379 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.684344 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bvxdz" event={"ID":"56e9f752-fd9a-4f05-8c86-fb30760212c3","Type":"ContainerStarted","Data":"05c5a68ae08b98b0229a0aba58aa657a5877ab01ea6145ba2323842bf6b6d700"} Jul 13 14:52:20.687668 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.687641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" event={"ID":"a4bac0d5aef81006c66c4c79f72062e4","Type":"ContainerStarted","Data":"244d304d88299cad37c1c90813d0868834120b86020b947c22046f84a7d12197"} Jul 13 14:52:20.690162 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.690134 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" event={"ID":"98260bfd566f14ae3c27b8cadea6e987","Type":"ContainerStarted","Data":"4e25ef07270b9b06dd043a261fb1236f484bbaccea055ba182dfb0006257c9fa"} Jul 13 14:52:20.691739 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.691712 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" event={"ID":"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b","Type":"ContainerStarted","Data":"31f6a71425b5a3999510b203d8e241cb9beabebd02cf5ac964afe9cf6dc70642"} Jul 13 14:52:20.694046 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.694018 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" event={"ID":"710e4f30-a65b-421f-a1fc-d738f21068ef","Type":"ContainerStarted","Data":"7dcef1fd4f047d66874535bea7f052225f9ffc73f07c4eea16c0f4bc81807ff1"} Jul 13 14:52:20.695455 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.695431 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerStarted","Data":"846d12bee5c6b22d9f6ad80d7f29af8a32690f91b379edb7224eb4f4c9768a62"} Jul 13 14:52:20.696987 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.696942 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m2k8m" event={"ID":"c601ac53-21fc-4809-80e6-b8cdc296e7e7","Type":"ContainerStarted","Data":"35eaa1ff3a8fea88ec997d7c2eac35424e940a59cd3dec47f912c3ee9cb35274"} Jul 13 14:52:20.698590 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.698567 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wf4g" event={"ID":"02703252-d08b-45ff-9716-08cbe91b83e1","Type":"ContainerStarted","Data":"44a882b3b9b252c86a787ca32c1666e4e61dd0a8dbc40bce77a280e20160612b"} Jul 13 14:52:20.700276 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.700254 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmz8z" event={"ID":"bc2e3881-4d9f-478e-afca-e481e662939b","Type":"ContainerStarted","Data":"991e483031716d65a3f4144c1323f520c5a4dba95ac907f1c17261ef674fc04a"} Jul 13 14:52:20.702378 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.702350 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"1a8a7be68f3425e03c2c8ea4edd793215541fa0b77210eab52f54a7ff0a671a3"} Jul 13 14:52:20.791884 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.791786 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:47:20 +0000 UTC" deadline="2028-01-05 08:36:38.959655072 +0000 UTC" Jul 13 14:52:20.791884 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.791865 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12977h44m18.167793954s" Jul 13 14:52:20.984025 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:20.983945 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:52:21.447623 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.447526 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:21.447623 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.447611 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:21.447884 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.447761 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:21.447884 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.447822 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:23.447803871 +0000 UTC m=+6.338478958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:21.448267 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.448247 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:21.448358 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.448306 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:23.448289045 +0000 UTC m=+6.338964136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:21.550240 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.550128 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:21.550240 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.550307 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:21.550871 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.550342 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:21.550871 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.550357 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:21.550871 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.550419 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:23.550397203 +0000 UTC m=+6.441072288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.675014 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.675144 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.675565 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.675669 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.675735 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:21.677538 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:21.675806 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:21.793122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.793018 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:47:20 +0000 UTC" deadline="2028-04-29 05:56:18.075462942 +0000 UTC" Jul 13 14:52:21.793122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:21.793073 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15735h3m56.282397151s" Jul 13 14:52:23.469430 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.469388 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:23.469927 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.469470 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:23.469927 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.469600 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:23.469927 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.469665 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:27.469644343 +0000 UTC m=+10.360319429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:23.470145 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.470123 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:23.470207 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.470185 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:27.470167853 +0000 UTC m=+10.360842942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:23.570628 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.570505 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:23.570798 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.570712 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:23.570798 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.570744 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:23.570798 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.570758 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:23.571005 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.570841 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:27.570819613 +0000 UTC m=+10.461494710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:23.673592 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.673557 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:23.673762 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.673699 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:23.674144 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.674124 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:23.674263 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.674238 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:23.674337 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:23.674317 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:23.674440 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:23.674418 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:25.673183 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:25.673109 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:25.673183 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:25.673113 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:25.673713 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:25.673289 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:25.673713 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:25.673367 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:25.673713 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:25.673506 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:25.673713 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:25.673605 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:27.507020 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.506982 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:27.507524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.507058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:27.507524 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.507202 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:27.507524 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.507261 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:35.507239406 +0000 UTC m=+18.397914503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:27.507524 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.507311 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:27.507524 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.507376 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:35.507358665 +0000 UTC m=+18.398033759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:27.608023 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.607366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:27.608395 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.608374 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:27.608514 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.608503 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:27.608598 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.608589 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:27.608759 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.608745 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:35.60871404 +0000 UTC m=+18.499389122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:27.673784 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.673643 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:27.673784 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.673744 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:27.673784 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:27.673789 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:27.674104 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.673821 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:27.674104 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.673885 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:27.674104 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:27.673960 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:29.677267 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:29.677231 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:29.677661 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:29.677364 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:29.677661 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:29.677391 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:29.677661 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:29.677482 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:29.677661 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:29.677508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:29.677661 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:29.677591 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:31.675950 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:31.675876 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:31.675950 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:31.675894 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:31.676442 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:31.675881 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:31.676442 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:31.676001 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:31.676442 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:31.676098 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:31.676442 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:31.676185 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:33.673629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:33.673496 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:33.673629 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:33.673630 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:33.674139 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:33.673702 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:33.674139 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:33.673845 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:33.674139 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:33.673866 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:33.674139 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:33.673955 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:35.567570 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.567529 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:35.568075 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.567607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:35.568075 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.567658 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:35.568075 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.567724 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:51.567703593 +0000 UTC m=+34.458378674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:35.568075 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.567724 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:35.568075 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.567776 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:52:51.567759702 +0000 UTC m=+34.458434797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:35.668539 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.668496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:35.668718 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.668695 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:35.668780 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.668724 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:35.668780 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.668737 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:35.668878 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.668816 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:52:51.668780719 +0000 UTC m=+34.559455815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:35.672514 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.672488 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:35.672626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.672519 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:35.672626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:35.672488 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:35.672626 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.672593 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:35.672768 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.672682 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:35.672832 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:35.672772 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:37.673612 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:37.673576 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:37.674130 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:37.673687 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:37.674130 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:37.673708 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:37.674130 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:37.673684 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:37.674130 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:37.673827 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:37.674130 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:37.673920 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:39.673190 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:39.673149 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:39.673190 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:39.673167 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:39.673698 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:39.673149 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:39.673698 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:39.673275 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:39.673698 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:39.673342 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:39.673698 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:39.673419 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:41.672900 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.672601 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:41.673534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.672614 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:41.673534 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:41.672979 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:41.673534 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:41.673039 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:41.673534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.672623 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:41.673534 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:41.673130 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:41.754962 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.754928 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" event={"ID":"710e4f30-a65b-421f-a1fc-d738f21068ef","Type":"ContainerStarted","Data":"c6df2216959379541ce72c2cff83a72d4df72e470e6076ac7640b3c747aec7e0"} Jul 13 14:52:41.756563 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.756521 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="7e52490e19ea9ec8a4c2f3261085307ca7d1889cfd3130c721a34b6861f8043e" exitCode=0 Jul 13 14:52:41.756687 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.756601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"7e52490e19ea9ec8a4c2f3261085307ca7d1889cfd3130c721a34b6861f8043e"} Jul 13 14:52:41.758070 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.758043 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m2k8m" event={"ID":"c601ac53-21fc-4809-80e6-b8cdc296e7e7","Type":"ContainerStarted","Data":"e2865536db2fcd749e27978e48c6e52c8ebca86fd4eebc02595ca8e3daa857ff"} Jul 13 14:52:41.759844 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.759821 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wf4g" event={"ID":"02703252-d08b-45ff-9716-08cbe91b83e1","Type":"ContainerStarted","Data":"ea48f41d15e8bf32be49b9f0dfab39398507024eef27e3e543005872acdd0513"} Jul 13 14:52:41.762653 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.762633 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:52:41.763141 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763116 2554 generic.go:356] "Generic (PLEG): container finished" podID="a425f617-f69e-48d4-9995-52d948d5c86e" containerID="15ed14c4c747adb228ecd96c1a4d97dbbf44a552d93e966feb56709b638061df" exitCode=1 Jul 13 14:52:41.763233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763144 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"cb31c01c993886e9167064391514fa45d64da445dbb9f21444dbccf7f4e6d23e"} Jul 13 14:52:41.763233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763224 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"02cf4c6b013d887beec6a5330e2db099f284246f2de0fff4f19c2577f21ab464"} Jul 13 14:52:41.763310 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763262 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"a35b9b28d2fe17b53a4f04377577c17e5b11d25e2277ba25f6c74e61fff0083e"} Jul 13 14:52:41.763310 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763301 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"628b51831b72600edb313663953732e4c2d4f6c0dbd5a1aac0f52cf2bbe3ca59"} Jul 13 14:52:41.763382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763338 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerDied","Data":"15ed14c4c747adb228ecd96c1a4d97dbbf44a552d93e966feb56709b638061df"} Jul 13 14:52:41.763382 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.763367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"09f6a18eb547012d53ed3ae7465f1e595650e2cfc4e7370befe3588864df2e3c"} Jul 13 14:52:41.764843 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.764824 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mppss" event={"ID":"72b450b5-9487-47a9-a267-e47ba75caa74","Type":"ContainerStarted","Data":"2a86ce57adfb64cd6250e02ae5756923a51946130cc5f2b80fc4ab2a5bf66e6f"} Jul 13 14:52:41.766289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.766267 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bvxdz" event={"ID":"56e9f752-fd9a-4f05-8c86-fb30760212c3","Type":"ContainerStarted","Data":"347aa824ee8d23709df88f8cfd00c78b48488895ab912682fe477794208c7043"} Jul 13 14:52:41.767910 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.767889 2554 generic.go:356] "Generic (PLEG): container finished" podID="a4bac0d5aef81006c66c4c79f72062e4" containerID="1935198a13a9b3836de3240c2bb2cf7a95a86bbb0bc00f62c37999a5810bef23" exitCode=0 Jul 13 14:52:41.768081 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.767927 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" event={"ID":"a4bac0d5aef81006c66c4c79f72062e4","Type":"ContainerDied","Data":"1935198a13a9b3836de3240c2bb2cf7a95a86bbb0bc00f62c37999a5810bef23"} Jul 13 14:52:41.769611 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.769589 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" event={"ID":"98260bfd566f14ae3c27b8cadea6e987","Type":"ContainerStarted","Data":"bc88ce3156bf0a6ea33f4e7f25098ff5ce54d892cb58fa58cf7b7d3865f0804b"} Jul 13 14:52:41.771071 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.771049 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" event={"ID":"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b","Type":"ContainerStarted","Data":"de0bf4235b2448b678ea38770377e76a199f2b55bf76f5679b3f6320388b81c1"} Jul 13 14:52:41.776641 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.776605 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x2q8r" podStartSLOduration=3.375580669 podStartE2EDuration="23.776593658s" totalImagesPullingTime="20.401012989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.776011207 +0000 UTC m=+24.666686322" watchObservedRunningTime="2026-07-13 14:52:41.776593658 +0000 UTC m=+24.667268759" Jul 13 14:52:41.792938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.792886 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wf4g" podStartSLOduration=11.460364889 podStartE2EDuration="23.792872092s" totalImagesPullingTime="12.332507203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.792548117 +0000 UTC m=+24.683223216" watchObservedRunningTime="2026-07-13 14:52:41.792872092 +0000 UTC m=+24.683547202" Jul 13 14:52:41.839546 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.839495 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m2k8m" podStartSLOduration=3.366014003 podStartE2EDuration="23.839473919s" totalImagesPullingTime="20.473459916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.817687776 +0000 UTC m=+24.708362878" watchObservedRunningTime="2026-07-13 14:52:41.839473919 +0000 UTC m=+24.730149021" Jul 13 14:52:41.854710 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.854664 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-241.ec2.internal" podStartSLOduration=21.854646805 podStartE2EDuration="21.854646805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.854595768 +0000 UTC m=+24.745270870" watchObservedRunningTime="2026-07-13 14:52:41.854646805 +0000 UTC m=+24.745321907" Jul 13 14:52:41.855000 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.854959 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mppss" podStartSLOduration=3.238335298 podStartE2EDuration="23.854946946s" totalImagesPullingTime="20.616611648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.840138341 +0000 UTC m=+24.730813444" watchObservedRunningTime="2026-07-13 14:52:41.854946946 +0000 UTC m=+24.745622081" Jul 13 14:52:41.885802 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:41.885756 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bvxdz" podStartSLOduration=3.407796849 podStartE2EDuration="23.885741982s" totalImagesPullingTime="20.477945133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:41.885583176 +0000 UTC m=+24.776258278" watchObservedRunningTime="2026-07-13 14:52:41.885741982 +0000 UTC m=+24.776417084" Jul 13 14:52:42.363778 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.363752 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 14:52:42.614612 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.614460 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T14:52:42.363774191Z","UUID":"5e154baf-bcd6-42b5-b77d-97db6179a057","Handler":null,"Name":"","Endpoint":""} Jul 13 14:52:42.616080 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.616055 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 14:52:42.616211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.616097 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 14:52:42.775980 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.775947 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" event={"ID":"a4bac0d5aef81006c66c4c79f72062e4","Type":"ContainerStarted","Data":"e57a344fdeb1e72bf321540b1dc4c8429869b98531eb1a224ba1b14840f740d2"} Jul 13 14:52:42.777938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.777883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" event={"ID":"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b","Type":"ContainerStarted","Data":"19feb9cb6ca436c94b0eefd28ed5a3c6cefa4b7c6be8c0c57e26739fbf1d9a08"} Jul 13 14:52:42.779387 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.779362 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmz8z" event={"ID":"bc2e3881-4d9f-478e-afca-e481e662939b","Type":"ContainerStarted","Data":"9d6d9c121c1a7d4b19b9cab7c64cf446eab2d6aacc1d6ade3a73812c6080fcdf"} Jul 13 14:52:42.794697 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.794647 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-241.ec2.internal" podStartSLOduration=22.794629455 podStartE2EDuration="22.794629455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:42.794480427 +0000 UTC m=+25.685155530" watchObservedRunningTime="2026-07-13 14:52:42.794629455 +0000 UTC m=+25.685304557" Jul 13 14:52:42.812169 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:42.812116 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xmz8z" podStartSLOduration=4.32187769 podStartE2EDuration="24.812097125s" totalImagesPullingTime="20.490219435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:42.811437429 +0000 UTC m=+25.702112531" watchObservedRunningTime="2026-07-13 14:52:42.812097125 +0000 UTC m=+25.702772614" Jul 13 14:52:43.467456 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.467424 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:43.468020 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.468005 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:43.672867 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.672840 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:43.672867 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.672858 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:43.673029 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:43.672958 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:43.673083 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:43.673025 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:43.673138 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.673089 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:43.673178 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:43.673165 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:43.785562 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.785478 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" event={"ID":"4cb516a0-9aca-42f4-b3b0-cdb1f0b54e1b","Type":"ContainerStarted","Data":"90d9547cd7b9a21cf2ff1f11d07ef3c372f76f71ee89866548ea0e0d8f0fa635"} Jul 13 14:52:43.789220 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.789202 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:52:43.789847 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.789824 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"5c5c4053e9b1664085406526c4ce6c83d0dc386c264ab709959f33ed211a5070"} Jul 13 14:52:43.790166 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.790149 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:43.790590 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:43.790573 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wf4g" Jul 13 14:52:44.820153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:44.819938 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgcq8" podStartSLOduration=4.171407889 podStartE2EDuration="26.819920066s" totalImagesPullingTime="22.648512177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:44.818919326 +0000 UTC m=+27.709594430" watchObservedRunningTime="2026-07-13 14:52:44.819920066 +0000 UTC m=+27.710595169" Jul 13 14:52:45.673161 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:45.673128 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:45.673350 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:45.673247 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:45.673350 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:45.673128 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:45.673452 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:45.673384 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:45.673452 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:45.673419 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:45.673514 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:45.673491 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:46.800410 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.800171 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:52:46.801243 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.800704 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"d38067e7fdeb8891de5908586e3623acc024f4654d90c548fd4399f0f3482fed"} Jul 13 14:52:46.801243 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.800941 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:46.801243 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.801175 2554 scope.go:117] "RemoveContainer" containerID="15ed14c4c747adb228ecd96c1a4d97dbbf44a552d93e966feb56709b638061df" Jul 13 14:52:46.802597 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.802577 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="246f0b79c608a7886c9cc32f791c651770a6df5b7cbb9e855b72b6b66070b340" exitCode=0 Jul 13 14:52:46.802691 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.802608 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"246f0b79c608a7886c9cc32f791c651770a6df5b7cbb9e855b72b6b66070b340"} Jul 13 14:52:46.817071 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:46.817047 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:47.673489 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.673458 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:47.673716 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.673531 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:47.673716 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:47.673542 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:47.673716 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:47.673655 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:47.673716 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.673707 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:47.673858 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:47.673767 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:47.808754 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.808332 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:52:47.808754 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.808691 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" event={"ID":"a425f617-f69e-48d4-9995-52d948d5c86e","Type":"ContainerStarted","Data":"470aab5e1a658886e82329bae57af7e333d82b2f8213b6259d5caf12f2217f39"} Jul 13 14:52:47.809063 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.809039 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:47.809214 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.809190 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:47.824812 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.824769 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:52:47.843328 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:47.843287 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" podStartSLOduration=9.364024534 podStartE2EDuration="29.843273658s" totalImagesPullingTime="20.479249124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:47.841872365 +0000 UTC m=+30.732547467" watchObservedRunningTime="2026-07-13 14:52:47.843273658 +0000 UTC m=+30.733948845" Jul 13 14:52:48.247093 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.247022 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6mfkf"] Jul 13 14:52:48.247227 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.247121 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:48.247290 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:48.247271 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:48.249736 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.249707 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2q2r"] Jul 13 14:52:48.249887 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.249826 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:48.249952 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:48.249907 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:48.254003 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.253979 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrlkt"] Jul 13 14:52:48.254127 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.254114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:48.254240 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:48.254220 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:48.812892 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.812858 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="64e07cbeca1815cf16a5439e397b8d81ff194468150ac3dc150d20cbd2595a3b" exitCode=0 Jul 13 14:52:48.813353 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:48.812938 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"64e07cbeca1815cf16a5439e397b8d81ff194468150ac3dc150d20cbd2595a3b"} Jul 13 14:52:49.673236 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:49.673203 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:49.673374 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:49.673344 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:50.673460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:50.673380 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:50.674020 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:50.673380 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:50.674020 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:50.673502 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:50.674020 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:50.673545 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:50.819422 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:50.819387 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="396eb8549e18bef9c48452ec8a1eca3d0f31d57fb8d6678a612949c2d1219fc6" exitCode=0 Jul 13 14:52:50.819576 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:50.819450 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"396eb8549e18bef9c48452ec8a1eca3d0f31d57fb8d6678a612949c2d1219fc6"} Jul 13 14:52:51.584981 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:51.584943 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:51.585143 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:51.585012 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:51.585143 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.585112 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:51.585244 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.585188 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret podName:b09ff867-1f8f-40fc-883f-7afd37b3b88b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:23.585168318 +0000 UTC m=+66.475843423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret") pod "global-pull-secret-syncer-6mfkf" (UID: "b09ff867-1f8f-40fc-883f-7afd37b3b88b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:52:51.585244 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.585123 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:51.585364 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.585255 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:23.585242373 +0000 UTC m=+66.475917458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:52:51.672907 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:51.672770 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:51.673067 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.672935 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:51.686213 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:51.686188 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:51.686624 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.686352 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:52:51.686624 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.686370 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:52:51.686624 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.686379 2554 projected.go:196] Error preparing data for projected volume kube-api-access-pzqw7 for pod openshift-network-diagnostics/network-check-target-g2q2r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:51.686624 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:51.686431 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7 podName:e780baeb-3c9c-4b4a-9c31-62e14d730b3a nodeName:}" failed. No retries permitted until 2026-07-13 14:53:23.686413635 +0000 UTC m=+66.577088716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzqw7" (UniqueName: "kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7") pod "network-check-target-g2q2r" (UID: "e780baeb-3c9c-4b4a-9c31-62e14d730b3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:52:52.672495 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:52.672411 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:52.672495 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:52.672441 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:52.672717 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:52.672537 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mfkf" podUID="b09ff867-1f8f-40fc-883f-7afd37b3b88b" Jul 13 14:52:52.672717 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:52.672628 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2q2r" podUID="e780baeb-3c9c-4b4a-9c31-62e14d730b3a" Jul 13 14:52:53.672883 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:53.672849 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:53.673381 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:53.673000 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrlkt" podUID="213edeac-f8f8-4ffc-864c-829a72c409f3" Jul 13 14:52:54.353983 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.353953 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-241.ec2.internal" event="NodeReady" Jul 13 14:52:54.354143 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.354098 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 14:52:54.396907 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.396873 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:52:54.399438 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.399417 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.402223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.402193 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 14:52:54.402590 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.402551 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 14:52:54.403244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.402758 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 14:52:54.403244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.403021 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-77ztl\"" Jul 13 14:52:54.409834 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.409817 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 14:52:54.417773 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.417754 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:52:54.507197 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507122 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507197 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507158 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v26k\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507381 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507269 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507381 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507310 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507381 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507357 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507510 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507382 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507510 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507445 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.507510 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.507478 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.535209 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.535181 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:54.608024 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.607985 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608039 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608119 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608169 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608211 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v26k\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.608422 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.608507 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:54.608456 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:52:54.608507 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:54.608470 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:52:54.608657 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:54.608535 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:52:55.108515209 +0000 UTC m=+37.999190290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:52:54.609114 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.609061 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.609370 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.609343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.612877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.612855 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.612877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.612868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.622456 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.622427 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v26k\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.623436 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.623411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:54.629978 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.629951 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:54.672820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.672778 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:52:54.672924 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.672777 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:52:54.675834 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.675814 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:52:54.675934 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.675903 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hw4cv\"" Jul 13 14:52:54.676002 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.675904 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 14:52:54.676178 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.676164 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 14:52:54.706735 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.706708 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:54.720781 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.720761 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zjf24"] Jul 13 14:52:54.723268 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.723253 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:54.726041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.726023 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 14:52:54.726133 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.726023 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4s4js\"" Jul 13 14:52:54.726303 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.726286 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 14:52:54.726368 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.726354 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 14:52:54.736312 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.736294 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zjf24"] Jul 13 14:52:54.739498 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.739479 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c9s7d"] Jul 13 14:52:54.741534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.741516 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:54.744070 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.744053 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mgh5\"" Jul 13 14:52:54.744189 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.744160 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 14:52:54.744273 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.744251 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 14:52:54.755100 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.755081 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9s7d"] Jul 13 14:52:54.776002 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.775976 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:54.830687 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.830656 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerStarted","Data":"ab207f06ae0eb281e60d3ba56b5168c0883c91220c9a74f1902e33e8743a8d7f"} Jul 13 14:52:54.911099 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:54.911220 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzht\" (UniqueName: \"kubernetes.io/projected/7d2dec88-ccda-4b33-85f7-870aece5497b-kube-api-access-wdzht\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:54.911220 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911205 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f2e019-aaa0-4042-99df-69b9fd53c892-tmp-dir\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:54.911320 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdst5\" (UniqueName: \"kubernetes.io/projected/d1f2e019-aaa0-4042-99df-69b9fd53c892-kube-api-access-fdst5\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:54.911320 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911252 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f2e019-aaa0-4042-99df-69b9fd53c892-config-volume\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:54.911320 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:54.911276 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2dec88-ccda-4b33-85f7-870aece5497b-cert\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.011883 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.011852 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzht\" (UniqueName: \"kubernetes.io/projected/7d2dec88-ccda-4b33-85f7-870aece5497b-kube-api-access-wdzht\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.012011 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.011927 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f2e019-aaa0-4042-99df-69b9fd53c892-tmp-dir\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.012011 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.011951 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdst5\" (UniqueName: \"kubernetes.io/projected/d1f2e019-aaa0-4042-99df-69b9fd53c892-kube-api-access-fdst5\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.012011 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.011973 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f2e019-aaa0-4042-99df-69b9fd53c892-config-volume\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.012011 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.011995 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2dec88-ccda-4b33-85f7-870aece5497b-cert\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.012206 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.012067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.012255 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.012214 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:52:55.012308 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.012283 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:55.512262862 +0000 UTC m=+38.402937941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:52:55.012383 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.012360 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1f2e019-aaa0-4042-99df-69b9fd53c892-tmp-dir\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.012729 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.012710 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f2e019-aaa0-4042-99df-69b9fd53c892-config-volume\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.014377 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.014357 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2dec88-ccda-4b33-85f7-870aece5497b-cert\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.020573 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.020524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdst5\" (UniqueName: \"kubernetes.io/projected/d1f2e019-aaa0-4042-99df-69b9fd53c892-kube-api-access-fdst5\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.021092 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.021076 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzht\" (UniqueName: \"kubernetes.io/projected/7d2dec88-ccda-4b33-85f7-870aece5497b-kube-api-access-wdzht\") pod \"ingress-canary-zjf24\" (UID: \"7d2dec88-ccda-4b33-85f7-870aece5497b\") " pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.040752 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.040732 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zjf24" Jul 13 14:52:55.112610 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.112456 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:55.112610 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.112605 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:52:55.112830 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.112621 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:52:55.112830 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.112673 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:52:56.112659019 +0000 UTC m=+39.003334102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:52:55.168544 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.168510 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zjf24"] Jul 13 14:52:55.171418 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:52:55.171388 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2dec88_ccda_4b33_85f7_870aece5497b.slice/crio-53d160bfb6fa1877463ee5076639ce7bcfd8ed2a0a4fe9470f943732c0527bf3 WatchSource:0}: Error finding container 53d160bfb6fa1877463ee5076639ce7bcfd8ed2a0a4fe9470f943732c0527bf3: Status 404 returned error can't find the container with id 53d160bfb6fa1877463ee5076639ce7bcfd8ed2a0a4fe9470f943732c0527bf3 Jul 13 14:52:55.473186 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.473079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:55.515122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.515092 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:55.515273 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.515209 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:52:55.515321 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:55.515287 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:56.5152737 +0000 UTC m=+39.405948780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:52:55.533917 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.533890 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:55.672940 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.672908 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:52:55.676056 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.676028 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bz4r7\"" Jul 13 14:52:55.676270 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.676254 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:52:55.834958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.834879 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zjf24" event={"ID":"7d2dec88-ccda-4b33-85f7-870aece5497b","Type":"ContainerStarted","Data":"53d160bfb6fa1877463ee5076639ce7bcfd8ed2a0a4fe9470f943732c0527bf3"} Jul 13 14:52:55.837831 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.837802 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="ab207f06ae0eb281e60d3ba56b5168c0883c91220c9a74f1902e33e8743a8d7f" exitCode=0 Jul 13 14:52:55.838007 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.837849 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"ab207f06ae0eb281e60d3ba56b5168c0883c91220c9a74f1902e33e8743a8d7f"} Jul 13 14:52:55.874177 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.874149 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:55.929695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:55.929664 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:56.121129 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:56.121041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:56.121276 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:56.121186 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:52:56.121276 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:56.121199 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:52:56.121276 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:56.121249 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:52:58.121235511 +0000 UTC m=+41.011910594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:52:56.525457 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:56.525374 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:56.525607 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:56.525568 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:52:56.525663 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:56.525637 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:58.525621495 +0000 UTC m=+41.416296575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:52:56.844050 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:56.843977 2554 generic.go:356] "Generic (PLEG): container finished" podID="160c96bd-e5f1-4bc8-9788-14dee3ac8af6" containerID="bbc104b8fd2e5493951cd4222c6947164d5100e352e34287bf85058a06d55553" exitCode=0 Jul 13 14:52:56.844421 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:56.844049 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerDied","Data":"bbc104b8fd2e5493951cd4222c6947164d5100e352e34287bf85058a06d55553"} Jul 13 14:52:57.847845 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:57.847636 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zjf24" event={"ID":"7d2dec88-ccda-4b33-85f7-870aece5497b","Type":"ContainerStarted","Data":"daa7d04c8b6f591aefc24397ae24798ce7661c6e2104c484ddc79b827818a2d7"} Jul 13 14:52:57.851126 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:57.851103 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" event={"ID":"160c96bd-e5f1-4bc8-9788-14dee3ac8af6","Type":"ContainerStarted","Data":"a6a4c41485b4f109ecb7f56d7d7b55742bbefaccd9f6a24b4faa977ca67b77f1"} Jul 13 14:52:57.875506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:57.875463 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zjf24" podStartSLOduration=1.656256198 podStartE2EDuration="3.8754491s" totalImagesPullingTime="2.219192902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:57.874741295 +0000 UTC m=+40.765416399" watchObservedRunningTime="2026-07-13 14:52:57.8754491 +0000 UTC m=+40.766124186" Jul 13 14:52:57.907424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:57.907392 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:57.910105 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:57.909774 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m7rh6" podStartSLOduration=8.662715245 podStartE2EDuration="39.909755923s" totalImagesPullingTime="31.247040678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:57.907617569 +0000 UTC m=+40.798292671" watchObservedRunningTime="2026-07-13 14:52:57.909755923 +0000 UTC m=+40.800431025" Jul 13 14:52:58.019476 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:58.019443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:58.137677 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:58.137588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:52:58.137862 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:58.137735 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:52:58.137862 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:58.137750 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:52:58.137862 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:58.137838 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:53:02.137818905 +0000 UTC m=+45.028494000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:52:58.540402 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:58.540368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:52:58.540554 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:58.540489 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:52:58.540554 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:52:58.540540 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:02.540526591 +0000 UTC m=+45.431201671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:52:59.267495 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:59.267461 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:59.346540 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:59.346510 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:59.502737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:59.502688 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:59.612411 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:52:59.612314 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:02.166307 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:02.166265 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:02.166670 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:02.166380 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:53:02.166670 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:02.166391 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:53:02.166670 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:02.166438 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:53:10.166424048 +0000 UTC m=+53.057099128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:53:02.569854 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:02.569822 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:02.570018 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:02.569978 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:53:02.570057 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:02.570042 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:10.570028366 +0000 UTC m=+53.460703446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:53:06.265264 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:06.265224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:06.364892 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:06.364864 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:10.224863 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.224825 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:10.225271 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.224941 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:53:10.225271 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.224953 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555977f595-c5jtk: secret "image-registry-tls" not found Jul 13 14:53:10.225271 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.225007 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls podName:32eb1b86-a88b-4406-ab8d-d50431ce4cce nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.224991791 +0000 UTC m=+69.115666871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls") pod "image-registry-555977f595-c5jtk" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce") : secret "image-registry-tls" not found Jul 13 14:53:10.627517 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.627479 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:10.627697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.627632 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:53:10.627697 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.627695 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls podName:d1f2e019-aaa0-4042-99df-69b9fd53c892 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.62767848 +0000 UTC m=+69.518353561 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls") pod "dns-default-c9s7d" (UID: "d1f2e019-aaa0-4042-99df-69b9fd53c892") : secret "dns-default-metrics-tls" not found Jul 13 14:53:10.703072 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.703042 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg"] Jul 13 14:53:10.720050 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.720018 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg"] Jul 13 14:53:10.720194 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.720134 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" Jul 13 14:53:10.724282 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.724255 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts"] Jul 13 14:53:10.724431 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.724410 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-24wl4\"" Jul 13 14:53:10.750807 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.749771 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5"] Jul 13 14:53:10.753013 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.752992 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" Jul 13 14:53:10.757801 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.756724 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.757801 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.757007 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k4b5d\"" Jul 13 14:53:10.757801 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.757192 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.765542 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.765516 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd"] Jul 13 14:53:10.765730 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.765708 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.768661 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.768642 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.768823 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.768804 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qgdhk\"" Jul 13 14:53:10.769389 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.769371 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.769389 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.769384 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 14:53:10.769900 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.769885 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 14:53:10.779587 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.779568 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts"] Jul 13 14:53:10.779664 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.779596 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd"] Jul 13 14:53:10.779664 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.779618 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5"] Jul 13 14:53:10.779732 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.779674 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.783041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.783022 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.783144 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.783125 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 13 14:53:10.783256 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.783230 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x2kjm\"" Jul 13 14:53:10.783414 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.783099 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 13 14:53:10.783571 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.783047 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.828303 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828269 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.828303 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828301 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3a4046-3553-4864-a26c-59f35f1f0da9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.828519 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828384 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwww\" (UniqueName: \"kubernetes.io/projected/bc3a4046-3553-4864-a26c-59f35f1f0da9-kube-api-access-sjwww\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.828519 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828444 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlm5m\" (UniqueName: \"kubernetes.io/projected/eca0983c-d818-4ffd-a300-f985896230d5-kube-api-access-zlm5m\") pod \"volume-data-source-validator-78495597f6-6jjts\" (UID: \"eca0983c-d818-4ffd-a300-f985896230d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" Jul 13 14:53:10.828519 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828475 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3a4046-3553-4864-a26c-59f35f1f0da9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.828643 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828530 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4rv\" (UniqueName: \"kubernetes.io/projected/190cd310-f8ad-4148-85d7-cd456b1d1552-kube-api-access-6j4rv\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.828643 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828566 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/190cd310-f8ad-4148-85d7-cd456b1d1552-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.828643 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.828584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qgz\" (UniqueName: \"kubernetes.io/projected/744dab42-4bfd-4291-9df6-2813af0a86e7-kube-api-access-s5qgz\") pod \"network-check-source-869dcfcdcd-r5rgg\" (UID: \"744dab42-4bfd-4291-9df6-2813af0a86e7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" Jul 13 14:53:10.832145 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.832122 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z"] Jul 13 14:53:10.851684 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.851665 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-j4cn7"] Jul 13 14:53:10.851809 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.851798 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:10.854588 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.854568 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 13 14:53:10.854678 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.854572 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 13 14:53:10.854918 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.854903 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-k6f2v\"" Jul 13 14:53:10.854958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.854907 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.854990 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.854956 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.877670 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.877620 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8km9s"] Jul 13 14:53:10.877767 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.877737 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.880438 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.880423 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.883586 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.883565 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.883678 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.883637 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fhz97\"" Jul 13 14:53:10.883749 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.883710 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 13 14:53:10.884162 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.884145 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 13 14:53:10.886758 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.886743 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 13 14:53:10.896263 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.896245 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66644b4cfd-4t6dc"] Jul 13 14:53:10.896364 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.896351 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:10.898962 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.898946 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 13 14:53:10.898962 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.898959 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 13 14:53:10.899244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.899088 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ktnj5\"" Jul 13 14:53:10.908263 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.908247 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z"] Jul 13 14:53:10.908337 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.908273 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8km9s"] Jul 13 14:53:10.908337 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.908288 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-j4cn7"] Jul 13 14:53:10.908337 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.908303 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66644b4cfd-4t6dc"] Jul 13 14:53:10.908424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.908348 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.910840 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.910822 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj"] Jul 13 14:53:10.911148 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911119 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.911335 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911319 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 13 14:53:10.911407 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911362 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 13 14:53:10.911407 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911368 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 13 14:53:10.911407 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911389 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 13 14:53:10.911594 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911575 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p27m8\"" Jul 13 14:53:10.911699 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.911619 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.926265 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.926247 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-mbl9k"] Jul 13 14:53:10.926379 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.926359 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:10.929206 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929182 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4rv\" (UniqueName: \"kubernetes.io/projected/190cd310-f8ad-4148-85d7-cd456b1d1552-kube-api-access-6j4rv\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.929388 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-default-certificate\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.929388 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929289 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:10.929388 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929382 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3a4046-3553-4864-a26c-59f35f1f0da9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929471 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9w4\" (UniqueName: \"kubernetes.io/projected/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-kube-api-access-cj9w4\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929511 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929545 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3a4046-3553-4864-a26c-59f35f1f0da9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929581 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlm5m\" (UniqueName: \"kubernetes.io/projected/eca0983c-d818-4ffd-a300-f985896230d5-kube-api-access-zlm5m\") pod \"volume-data-source-validator-78495597f6-6jjts\" (UID: \"eca0983c-d818-4ffd-a300-f985896230d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" Jul 13 14:53:10.929617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929613 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-config\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929649 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e632f3ac-8807-4f49-b083-2b41f69e6890-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929688 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/190cd310-f8ad-4148-85d7-cd456b1d1552-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929714 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929749 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929718 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qgz\" (UniqueName: \"kubernetes.io/projected/744dab42-4bfd-4291-9df6-2813af0a86e7-kube-api-access-s5qgz\") pod \"network-check-source-869dcfcdcd-r5rgg\" (UID: \"744dab42-4bfd-4291-9df6-2813af0a86e7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" Jul 13 14:53:10.929970 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929958 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930048 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930067 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2962n\"" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930084 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zj7\" (UniqueName: \"kubernetes.io/projected/254d637f-eb4a-4b2e-ac6b-9ac05474709b-kube-api-access-j4zj7\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930133 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-stats-auth\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-tmp\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwww\" (UniqueName: \"kubernetes.io/projected/bc3a4046-3553-4864-a26c-59f35f1f0da9-kube-api-access-sjwww\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.930250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930233 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-snapshots\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.930616 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930271 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ca4899-57dd-4122-a6aa-031e069d9f82-serving-cert\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.930616 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930351 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.930616 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930390 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7ls\" (UniqueName: \"kubernetes.io/projected/35ca4899-57dd-4122-a6aa-031e069d9f82-kube-api-access-pb7ls\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:10.930616 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.930466 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:10.930616 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:10.930538 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.430513687 +0000 UTC m=+54.321188781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:10.930920 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.929651 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.930920 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.930463 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3a4046-3553-4864-a26c-59f35f1f0da9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.931544 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.931489 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/190cd310-f8ad-4148-85d7-cd456b1d1552-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.934679 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.934658 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3a4046-3553-4864-a26c-59f35f1f0da9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.940454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.940427 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4rv\" (UniqueName: \"kubernetes.io/projected/190cd310-f8ad-4148-85d7-cd456b1d1552-kube-api-access-6j4rv\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:10.944944 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.944923 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwww\" (UniqueName: \"kubernetes.io/projected/bc3a4046-3553-4864-a26c-59f35f1f0da9-kube-api-access-sjwww\") pod \"kube-storage-version-migrator-operator-799cb7f958-q6bnd\" (UID: \"bc3a4046-3553-4864-a26c-59f35f1f0da9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:10.947423 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.947390 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlm5m\" (UniqueName: \"kubernetes.io/projected/eca0983c-d818-4ffd-a300-f985896230d5-kube-api-access-zlm5m\") pod \"volume-data-source-validator-78495597f6-6jjts\" (UID: \"eca0983c-d818-4ffd-a300-f985896230d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" Jul 13 14:53:10.948244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.948226 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qgz\" (UniqueName: \"kubernetes.io/projected/744dab42-4bfd-4291-9df6-2813af0a86e7-kube-api-access-s5qgz\") pod \"network-check-source-869dcfcdcd-r5rgg\" (UID: \"744dab42-4bfd-4291-9df6-2813af0a86e7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" Jul 13 14:53:10.954520 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.954500 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj"] Jul 13 14:53:10.954616 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.954542 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mbl9k"] Jul 13 14:53:10.954616 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.954609 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:10.957431 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.957407 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 13 14:53:10.957518 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.957452 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:10.957518 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.957461 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v84xp\"" Jul 13 14:53:10.957629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.957409 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 13 14:53:10.957629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.957547 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 13 14:53:10.962232 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.962213 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 13 14:53:10.976095 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:10.976073 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.029856 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.029835 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" Jul 13 14:53:11.031083 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031069 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-stats-auth\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.031153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031092 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-tmp\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031111 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-config\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.031153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031128 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-snapshots\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031302 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031173 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ca4899-57dd-4122-a6aa-031e069d9f82-serving-cert\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031302 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031393 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7ls\" (UniqueName: \"kubernetes.io/projected/35ca4899-57dd-4122-a6aa-031e069d9f82-kube-api-access-pb7ls\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031393 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031385 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-trusted-ca\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.031543 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.031626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031558 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-default-certificate\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.031626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031587 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-serving-cert\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.031626 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031622 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.031768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-tmp\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031660 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031721 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9w4\" (UniqueName: \"kubernetes.io/projected/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-kube-api-access-cj9w4\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.031768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031740 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35ca4899-57dd-4122-a6aa-031e069d9f82-snapshots\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.031768 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031745 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031807 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031825 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmncd\" (UniqueName: \"kubernetes.io/projected/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-kube-api-access-xmncd\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031869 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-config\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031906 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pltr\" (UniqueName: \"kubernetes.io/projected/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-kube-api-access-7pltr\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031938 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e632f3ac-8807-4f49-b083-2b41f69e6890-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.031990 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:11.032026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.032022 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.032346 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.032078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zj7\" (UniqueName: \"kubernetes.io/projected/254d637f-eb4a-4b2e-ac6b-9ac05474709b-kube-api-access-j4zj7\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.032346 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.032156 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.532137789 +0000 UTC m=+54.422812876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:11.032346 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.032224 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:11.032346 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.032279 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.532266667 +0000 UTC m=+54.422941753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:11.032346 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.032344 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:53:11.032589 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.032377 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.532365634 +0000 UTC m=+54.423040719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : secret "router-metrics-certs-default" not found Jul 13 14:53:11.032589 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.032482 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35ca4899-57dd-4122-a6aa-031e069d9f82-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.032589 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.032554 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e632f3ac-8807-4f49-b083-2b41f69e6890-nginx-conf\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:11.033017 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.032997 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-config\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.034023 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.033996 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-stats-auth\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.034218 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.034203 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ca4899-57dd-4122-a6aa-031e069d9f82-serving-cert\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.034273 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.034232 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.034341 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.034327 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-default-certificate\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.047499 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.047478 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zj7\" (UniqueName: \"kubernetes.io/projected/254d637f-eb4a-4b2e-ac6b-9ac05474709b-kube-api-access-j4zj7\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.051326 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.051307 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9w4\" (UniqueName: \"kubernetes.io/projected/c26e4564-e76c-4b18-81ee-cb396ab0d1c3-kube-api-access-cj9w4\") pod \"service-ca-operator-7695fc6c8c-qbz7z\" (UID: \"c26e4564-e76c-4b18-81ee-cb396ab0d1c3\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.058042 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.058021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7ls\" (UniqueName: \"kubernetes.io/projected/35ca4899-57dd-4122-a6aa-031e069d9f82-kube-api-access-pb7ls\") pod \"insights-operator-68d9dd94cf-j4cn7\" (UID: \"35ca4899-57dd-4122-a6aa-031e069d9f82\") " pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.066706 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.066690 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" Jul 13 14:53:11.089502 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.089483 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" Jul 13 14:53:11.114233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.114199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.133584 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.133520 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmncd\" (UniqueName: \"kubernetes.io/projected/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-kube-api-access-xmncd\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.134199 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pltr\" (UniqueName: \"kubernetes.io/projected/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-kube-api-access-7pltr\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.134326 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-config\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.134384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-trusted-ca\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.134415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.134444 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-serving-cert\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.135513 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 14:53:11.136212 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.135599 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls podName:a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.63557735 +0000 UTC m=+54.526252433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls") pod "cluster-samples-operator-65db858894-s9jxj" (UID: "a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca") : secret "samples-operator-tls" not found Jul 13 14:53:11.138457 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.137646 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-trusted-ca\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.138457 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.138394 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-config\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.139334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.139262 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-serving-cert\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.145649 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.145592 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmncd\" (UniqueName: \"kubernetes.io/projected/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-kube-api-access-xmncd\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.146333 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.146276 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pltr\" (UniqueName: \"kubernetes.io/projected/c8b6e5d5-b197-4dfc-a471-496cbd3d24ee-kube-api-access-7pltr\") pod \"console-operator-7866497494-mbl9k\" (UID: \"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee\") " pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.160159 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.160136 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" Jul 13 14:53:11.185899 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.185874 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" Jul 13 14:53:11.213887 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.213855 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg"] Jul 13 14:53:11.214528 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.214498 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744dab42_4bfd_4291_9df6_2813af0a86e7.slice/crio-6b42d14e6bb60a5e50b5626da113eb12f00bafe1dc19839101439c8a2ddcc254 WatchSource:0}: Error finding container 6b42d14e6bb60a5e50b5626da113eb12f00bafe1dc19839101439c8a2ddcc254: Status 404 returned error can't find the container with id 6b42d14e6bb60a5e50b5626da113eb12f00bafe1dc19839101439c8a2ddcc254 Jul 13 14:53:11.219079 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.218962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.264021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.263819 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:11.264432 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.264364 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts"] Jul 13 14:53:11.270330 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.270304 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca0983c_d818_4ffd_a300_f985896230d5.slice/crio-f9c5e68f7b9e8541af80cdaa9409da074ea08fdf2c0d649bdf8313e77cd8fc44 WatchSource:0}: Error finding container f9c5e68f7b9e8541af80cdaa9409da074ea08fdf2c0d649bdf8313e77cd8fc44: Status 404 returned error can't find the container with id f9c5e68f7b9e8541af80cdaa9409da074ea08fdf2c0d649bdf8313e77cd8fc44 Jul 13 14:53:11.287508 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.287456 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd"] Jul 13 14:53:11.293739 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.293707 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3a4046_3553_4864_a26c_59f35f1f0da9.slice/crio-d26fe4a1cc6d282e181becd787dc38acdac6b5ad4d967527548aa4fd2bf39ec5 WatchSource:0}: Error finding container d26fe4a1cc6d282e181becd787dc38acdac6b5ad4d967527548aa4fd2bf39ec5: Status 404 returned error can't find the container with id d26fe4a1cc6d282e181becd787dc38acdac6b5ad4d967527548aa4fd2bf39ec5 Jul 13 14:53:11.314285 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.314200 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.344809 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.344772 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z"] Jul 13 14:53:11.347088 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.347063 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26e4564_e76c_4b18_81ee_cb396ab0d1c3.slice/crio-22ba4c926406c687060e27b3c85599c0d7afd7ff1efa3b217588c3a052128297 WatchSource:0}: Error finding container 22ba4c926406c687060e27b3c85599c0d7afd7ff1efa3b217588c3a052128297: Status 404 returned error can't find the container with id 22ba4c926406c687060e27b3c85599c0d7afd7ff1efa3b217588c3a052128297 Jul 13 14:53:11.353974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.353951 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-j4cn7"] Jul 13 14:53:11.357545 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.357508 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ca4899_57dd_4122_a6aa_031e069d9f82.slice/crio-5efcdcc5bbd7ace86837409cbed9c330a77d408cdfd8fe49e963fd52153d167e WatchSource:0}: Error finding container 5efcdcc5bbd7ace86837409cbed9c330a77d408cdfd8fe49e963fd52153d167e: Status 404 returned error can't find the container with id 5efcdcc5bbd7ace86837409cbed9c330a77d408cdfd8fe49e963fd52153d167e Jul 13 14:53:11.401996 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.401936 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mbl9k"] Jul 13 14:53:11.404879 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:11.404846 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b6e5d5_b197_4dfc_a471_496cbd3d24ee.slice/crio-13046006c0312e083e3d30ba94c47d571188b197d60a8f8b632a78f484dd80f2 WatchSource:0}: Error finding container 13046006c0312e083e3d30ba94c47d571188b197d60a8f8b632a78f484dd80f2: Status 404 returned error can't find the container with id 13046006c0312e083e3d30ba94c47d571188b197d60a8f8b632a78f484dd80f2 Jul 13 14:53:11.437573 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.437550 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:11.437721 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.437701 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:11.437770 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.437760 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:12.437745739 +0000 UTC m=+55.328420819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:11.538603 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.538573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:11.538603 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.538607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.538832 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.538739 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:11.538887 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.538830 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:11.538887 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.538846 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:12.53882572 +0000 UTC m=+55.429500801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:11.538887 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.538864 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:53:11.539045 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.538901 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:12.538890487 +0000 UTC m=+55.429565567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : secret "router-metrics-certs-default" not found Jul 13 14:53:11.539045 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.538917 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:12.538910948 +0000 UTC m=+55.429586028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:11.639959 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.639925 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:11.640093 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.640061 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 14:53:11.640146 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:11.640119 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls podName:a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca nodeName:}" failed. No retries permitted until 2026-07-13 14:53:12.640104821 +0000 UTC m=+55.530779901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls") pod "cluster-samples-operator-65db858894-s9jxj" (UID: "a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca") : secret "samples-operator-tls" not found Jul 13 14:53:11.883132 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.883093 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" event={"ID":"744dab42-4bfd-4291-9df6-2813af0a86e7","Type":"ContainerStarted","Data":"6b42d14e6bb60a5e50b5626da113eb12f00bafe1dc19839101439c8a2ddcc254"} Jul 13 14:53:11.884302 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.884267 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" event={"ID":"35ca4899-57dd-4122-a6aa-031e069d9f82","Type":"ContainerStarted","Data":"5efcdcc5bbd7ace86837409cbed9c330a77d408cdfd8fe49e963fd52153d167e"} Jul 13 14:53:11.885467 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.885433 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mbl9k" event={"ID":"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee","Type":"ContainerStarted","Data":"13046006c0312e083e3d30ba94c47d571188b197d60a8f8b632a78f484dd80f2"} Jul 13 14:53:11.886535 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.886509 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" event={"ID":"c26e4564-e76c-4b18-81ee-cb396ab0d1c3","Type":"ContainerStarted","Data":"22ba4c926406c687060e27b3c85599c0d7afd7ff1efa3b217588c3a052128297"} Jul 13 14:53:11.887632 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.887609 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" event={"ID":"bc3a4046-3553-4864-a26c-59f35f1f0da9","Type":"ContainerStarted","Data":"d26fe4a1cc6d282e181becd787dc38acdac6b5ad4d967527548aa4fd2bf39ec5"} Jul 13 14:53:11.888668 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:11.888649 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" event={"ID":"eca0983c-d818-4ffd-a300-f985896230d5","Type":"ContainerStarted","Data":"f9c5e68f7b9e8541af80cdaa9409da074ea08fdf2c0d649bdf8313e77cd8fc44"} Jul 13 14:53:12.449036 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:12.448992 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:12.449501 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.449160 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:12.449501 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.449227 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:14.449207509 +0000 UTC m=+57.339882608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:12.550481 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:12.550533 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:12.550696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.550944 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.551001 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:14.550982984 +0000 UTC m=+57.441658068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.551413 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:53:12.551485 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.551459 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:14.551445127 +0000 UTC m=+57.442120211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : secret "router-metrics-certs-default" not found Jul 13 14:53:12.552096 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.552066 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:14.552045183 +0000 UTC m=+57.442720265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:12.651755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:12.651717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:12.652072 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.652052 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 14:53:12.652160 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:12.652124 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls podName:a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca nodeName:}" failed. No retries permitted until 2026-07-13 14:53:14.652105194 +0000 UTC m=+57.542780276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls") pod "cluster-samples-operator-65db858894-s9jxj" (UID: "a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca") : secret "samples-operator-tls" not found Jul 13 14:53:13.844553 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:13.844507 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:13.946121 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:13.946088 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:14.470960 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:14.470925 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:14.471117 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.471076 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:14.471162 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.471154 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.471138496 +0000 UTC m=+61.361813576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:14.571890 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:14.571853 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:14.572051 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:14.571918 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:14.572051 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:14.571949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:14.572051 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.572024 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:14.572181 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.572066 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:53:14.572181 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.572089 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.572066469 +0000 UTC m=+61.462741557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:14.572181 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.572116 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.572105595 +0000 UTC m=+61.462780680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:14.572181 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.572133 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.572122456 +0000 UTC m=+61.462797540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : secret "router-metrics-certs-default" not found Jul 13 14:53:14.672935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:14.672761 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:14.672935 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.672902 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 14:53:14.673174 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:14.672990 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls podName:a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.672969424 +0000 UTC m=+61.563644509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls") pod "cluster-samples-operator-65db858894-s9jxj" (UID: "a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca") : secret "samples-operator-tls" not found Jul 13 14:53:16.594231 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.594204 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:16.913326 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.913293 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" event={"ID":"35ca4899-57dd-4122-a6aa-031e069d9f82","Type":"ContainerStarted","Data":"4f575d5fe79130613c4ccced3a7adebe8f83e597b78f678b1122572ceeb60c28"} Jul 13 14:53:16.915135 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.915109 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/0.log" Jul 13 14:53:16.915274 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.915155 2554 generic.go:356] "Generic (PLEG): container finished" podID="c8b6e5d5-b197-4dfc-a471-496cbd3d24ee" containerID="ca67077257ca1b7dde31c5b56e1073d45f30b643da6b7eb764ac83cedcf261dd" exitCode=255 Jul 13 14:53:16.915274 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.915228 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mbl9k" event={"ID":"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee","Type":"ContainerDied","Data":"ca67077257ca1b7dde31c5b56e1073d45f30b643da6b7eb764ac83cedcf261dd"} Jul 13 14:53:16.915424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.915406 2554 scope.go:117] "RemoveContainer" containerID="ca67077257ca1b7dde31c5b56e1073d45f30b643da6b7eb764ac83cedcf261dd" Jul 13 14:53:16.917178 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.917155 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" event={"ID":"c26e4564-e76c-4b18-81ee-cb396ab0d1c3","Type":"ContainerStarted","Data":"69625a43fcfddbfb30bd540b6292de65cf34f3fef274b3d015039bf6deb56335"} Jul 13 14:53:16.919110 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.919088 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" event={"ID":"bc3a4046-3553-4864-a26c-59f35f1f0da9","Type":"ContainerStarted","Data":"4302629f027ef4908492ffcd6dd93ea7cb348e275be313e2378a1cd37d258514"} Jul 13 14:53:16.920977 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.920955 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" event={"ID":"eca0983c-d818-4ffd-a300-f985896230d5","Type":"ContainerStarted","Data":"a3dff76c0c19820fe2b387e2cb0c730869fa5c8d0cfd58e8378bceaf94d1c17b"} Jul 13 14:53:16.922445 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.922424 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" event={"ID":"744dab42-4bfd-4291-9df6-2813af0a86e7","Type":"ContainerStarted","Data":"14af38e885d6a793e61213d3d3019e58efc52be45db8734a0d4dd38e2a5d878a"} Jul 13 14:53:16.932318 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.932269 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" podStartSLOduration=1.888593213 podStartE2EDuration="6.932253252s" totalImagesPullingTime="5.043660039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:16.931309956 +0000 UTC m=+59.821985059" watchObservedRunningTime="2026-07-13 14:53:16.932253252 +0000 UTC m=+59.822928336" Jul 13 14:53:16.968506 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.968460 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r5rgg" podStartSLOduration=1.7763686029999999 podStartE2EDuration="6.968439669s" totalImagesPullingTime="5.192071066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:16.948750151 +0000 UTC m=+59.839425257" watchObservedRunningTime="2026-07-13 14:53:16.968439669 +0000 UTC m=+59.859114772" Jul 13 14:53:16.969725 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:16.969270 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" podStartSLOduration=1.870977101 podStartE2EDuration="6.969257486s" totalImagesPullingTime="5.098280385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:16.967796999 +0000 UTC m=+59.858472094" watchObservedRunningTime="2026-07-13 14:53:16.969257486 +0000 UTC m=+59.859932590" Jul 13 14:53:17.012856 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.012028 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-6jjts" podStartSLOduration=1.899692232 podStartE2EDuration="7.012008345s" totalImagesPullingTime="5.112316113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.011882188 +0000 UTC m=+59.902557287" watchObservedRunningTime="2026-07-13 14:53:17.012008345 +0000 UTC m=+59.902683449" Jul 13 14:53:17.148234 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.148198 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.247256 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.247181 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.458773 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.458740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.488439 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.488386 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" podStartSLOduration=2.437549985 podStartE2EDuration="7.488364831s" totalImagesPullingTime="5.050814846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.035073012 +0000 UTC m=+59.925748115" watchObservedRunningTime="2026-07-13 14:53:17.488364831 +0000 UTC m=+60.379039933" Jul 13 14:53:17.489846 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.489819 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66"] Jul 13 14:53:17.492291 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.492272 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" Jul 13 14:53:17.494743 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.494717 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 13 14:53:17.495145 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.495125 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 13 14:53:17.495844 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.495830 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4zmzc\"" Jul 13 14:53:17.503831 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.503810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66"] Jul 13 14:53:17.547731 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.547690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.599451 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.599419 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8r7\" (UniqueName: \"kubernetes.io/projected/d746f33c-9114-40c8-b256-1bb5d0d36485-kube-api-access-lb8r7\") pod \"migrator-7f59b54f6c-w6k66\" (UID: \"d746f33c-9114-40c8-b256-1bb5d0d36485\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" Jul 13 14:53:17.621436 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.621396 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.700133 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.700096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8r7\" (UniqueName: \"kubernetes.io/projected/d746f33c-9114-40c8-b256-1bb5d0d36485-kube-api-access-lb8r7\") pod \"migrator-7f59b54f6c-w6k66\" (UID: \"d746f33c-9114-40c8-b256-1bb5d0d36485\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" Jul 13 14:53:17.709331 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.709298 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8r7\" (UniqueName: \"kubernetes.io/projected/d746f33c-9114-40c8-b256-1bb5d0d36485-kube-api-access-lb8r7\") pod \"migrator-7f59b54f6c-w6k66\" (UID: \"d746f33c-9114-40c8-b256-1bb5d0d36485\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" Jul 13 14:53:17.736695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.736645 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.797752 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.797717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.803533 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.803509 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" Jul 13 14:53:17.927021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.926990 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 14:53:17.927405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.927391 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/0.log" Jul 13 14:53:17.927488 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.927424 2554 generic.go:356] "Generic (PLEG): container finished" podID="c8b6e5d5-b197-4dfc-a471-496cbd3d24ee" containerID="436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25" exitCode=255 Jul 13 14:53:17.927540 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.927523 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mbl9k" event={"ID":"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee","Type":"ContainerDied","Data":"436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25"} Jul 13 14:53:17.927593 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.927580 2554 scope.go:117] "RemoveContainer" containerID="ca67077257ca1b7dde31c5b56e1073d45f30b643da6b7eb764ac83cedcf261dd" Jul 13 14:53:17.927841 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.927825 2554 scope.go:117] "RemoveContainer" containerID="436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25" Jul 13 14:53:17.928043 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:17.928011 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mbl9k_openshift-console-operator(c8b6e5d5-b197-4dfc-a471-496cbd3d24ee)\"" pod="openshift-console-operator/console-operator-7866497494-mbl9k" podUID="c8b6e5d5-b197-4dfc-a471-496cbd3d24ee" Jul 13 14:53:17.930625 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:17.930604 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66"] Jul 13 14:53:17.933556 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:17.933536 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd746f33c_9114_40c8_b256_1bb5d0d36485.slice/crio-bce26a7bc70f0d56887637a62a551a8b96f97090a9c493eeaa2cbf72ec512a3a WatchSource:0}: Error finding container bce26a7bc70f0d56887637a62a551a8b96f97090a9c493eeaa2cbf72ec512a3a: Status 404 returned error can't find the container with id bce26a7bc70f0d56887637a62a551a8b96f97090a9c493eeaa2cbf72ec512a3a Jul 13 14:53:18.026637 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.026603 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9svjz"] Jul 13 14:53:18.029417 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.029398 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.031721 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.031702 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 14:53:18.032022 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.031998 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 14:53:18.032538 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.032523 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ghzxf\"" Jul 13 14:53:18.042162 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.042139 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9svjz"] Jul 13 14:53:18.107677 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.107612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.107677 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.107661 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2gh\" (UniqueName: \"kubernetes.io/projected/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-api-access-hh2gh\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.107866 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.107723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-crio-socket\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.107866 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.107751 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-data-volume\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.107929 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.107886 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209246 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209216 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2gh\" (UniqueName: \"kubernetes.io/projected/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-api-access-hh2gh\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209403 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209287 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-crio-socket\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209403 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209326 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-data-volume\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209403 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209361 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-crio-socket\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209500 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209419 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209500 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209471 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209612 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.209592 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:53:18.209666 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.209656 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.709641741 +0000 UTC m=+61.600316824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : secret "insights-runtime-extractor-tls" not found Jul 13 14:53:18.209714 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.209683 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-data-volume\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.209714 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.209594 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:53:18.209848 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.209753 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:18.709739807 +0000 UTC m=+61.600414886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : configmap "kube-rbac-proxy" not found Jul 13 14:53:18.218028 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.218000 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2gh\" (UniqueName: \"kubernetes.io/projected/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-api-access-hh2gh\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.511983 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.511954 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:18.512131 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.512106 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:18.512185 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.512174 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.512159702 +0000 UTC m=+69.402834781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:18.613218 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.613177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.613265 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.613295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.613358 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.613335582 +0000 UTC m=+69.504010677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.613421 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.613470 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.613454945 +0000 UTC m=+69.504130031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : secret "router-metrics-certs-default" not found Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.613489 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:18.613645 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.613524 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.613513535 +0000 UTC m=+69.504188620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:18.714583 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.714545 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.714742 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.714596 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:18.714742 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714663 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:53:18.714742 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714725 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:19.714708383 +0000 UTC m=+62.605383463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : configmap "kube-rbac-proxy" not found Jul 13 14:53:18.714940 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714774 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 14:53:18.714940 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.714802 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:18.714940 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714853 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls podName:a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca nodeName:}" failed. No retries permitted until 2026-07-13 14:53:26.714831942 +0000 UTC m=+69.605507037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls") pod "cluster-samples-operator-65db858894-s9jxj" (UID: "a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca") : secret "samples-operator-tls" not found Jul 13 14:53:18.715090 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714939 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:53:18.715090 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.714988 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:19.714972308 +0000 UTC m=+62.605647412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : secret "insights-runtime-extractor-tls" not found Jul 13 14:53:18.932959 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.932828 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" event={"ID":"d746f33c-9114-40c8-b256-1bb5d0d36485","Type":"ContainerStarted","Data":"bce26a7bc70f0d56887637a62a551a8b96f97090a9c493eeaa2cbf72ec512a3a"} Jul 13 14:53:18.935202 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.935177 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 14:53:18.935602 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:18.935582 2554 scope.go:117] "RemoveContainer" containerID="436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25" Jul 13 14:53:18.935828 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:18.935781 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mbl9k_openshift-console-operator(c8b6e5d5-b197-4dfc-a471-496cbd3d24ee)\"" pod="openshift-console-operator/console-operator-7866497494-mbl9k" podUID="c8b6e5d5-b197-4dfc-a471-496cbd3d24ee" Jul 13 14:53:19.219358 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.219286 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:19.351148 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.351122 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:19.725331 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.725294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:19.725737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.725345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:19.725737 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:19.725450 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:53:19.725737 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:19.725511 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:21.72549534 +0000 UTC m=+64.616170420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : secret "insights-runtime-extractor-tls" not found Jul 13 14:53:19.725737 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:19.725514 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:53:19.725737 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:19.725578 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:21.72555646 +0000 UTC m=+64.616231561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : configmap "kube-rbac-proxy" not found Jul 13 14:53:19.831443 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.831414 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khk5x" Jul 13 14:53:19.939244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.939212 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" event={"ID":"d746f33c-9114-40c8-b256-1bb5d0d36485","Type":"ContainerStarted","Data":"272dc7a7f6c1a73bf771a0482ae104169ceb5a42f34e4e54fb4a92b533b11da0"} Jul 13 14:53:19.939402 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.939260 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" event={"ID":"d746f33c-9114-40c8-b256-1bb5d0d36485","Type":"ContainerStarted","Data":"b671fd80dc318a4bfa06f960e9ed68d53f8b46abbbd848369f59ccea4a165499"} Jul 13 14:53:19.981556 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:19.981502 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-w6k66" podStartSLOduration=1.6307412380000001 podStartE2EDuration="2.981486583s" totalImagesPullingTime="1.350745345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:19.975194241 +0000 UTC m=+62.865869353" watchObservedRunningTime="2026-07-13 14:53:19.981486583 +0000 UTC m=+62.872161684" Jul 13 14:53:20.035597 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.035569 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m2k8m_c601ac53-21fc-4809-80e6-b8cdc296e7e7/dns-node-resolver/0.log" Jul 13 14:53:20.077454 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.077420 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4qt4p"] Jul 13 14:53:20.097952 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.097885 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4qt4p"] Jul 13 14:53:20.098071 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.097979 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.103806 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.103755 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 14:53:20.103951 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.103821 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 14:53:20.104183 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.104102 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g5z5v\"" Jul 13 14:53:20.104234 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.104200 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 14:53:20.104952 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.104936 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 14:53:20.143394 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.143369 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:20.231198 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.231176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-cabundle\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.231302 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.231239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6cg\" (UniqueName: \"kubernetes.io/projected/dbbc9a37-7489-4812-9f73-43fae8824d1d-kube-api-access-js6cg\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.231342 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.231307 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-key\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.332679 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.332651 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-cabundle\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.332875 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.332743 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6cg\" (UniqueName: \"kubernetes.io/projected/dbbc9a37-7489-4812-9f73-43fae8824d1d-kube-api-access-js6cg\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.332875 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.332771 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-key\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.333329 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.333311 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-cabundle\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.335201 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.335179 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbbc9a37-7489-4812-9f73-43fae8824d1d-signing-key\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.357134 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.357060 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6cg\" (UniqueName: \"kubernetes.io/projected/dbbc9a37-7489-4812-9f73-43fae8824d1d-kube-api-access-js6cg\") pod \"service-ca-79f68b9f4f-4qt4p\" (UID: \"dbbc9a37-7489-4812-9f73-43fae8824d1d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.407087 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.407059 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" Jul 13 14:53:20.573875 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.573841 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4qt4p"] Jul 13 14:53:20.577349 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:20.577322 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbc9a37_7489_4812_9f73_43fae8824d1d.slice/crio-a0b078d1a4a9b93c9426f156903e3273ea9cf777ca99dea992ba3a032c3da27c WatchSource:0}: Error finding container a0b078d1a4a9b93c9426f156903e3273ea9cf777ca99dea992ba3a032c3da27c: Status 404 returned error can't find the container with id a0b078d1a4a9b93c9426f156903e3273ea9cf777ca99dea992ba3a032c3da27c Jul 13 14:53:20.944009 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.943925 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" event={"ID":"dbbc9a37-7489-4812-9f73-43fae8824d1d","Type":"ContainerStarted","Data":"337c078af22a66d50fa0feb19216046d3002add1236552986fba3f926dc0347c"} Jul 13 14:53:20.944009 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:20.943981 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" event={"ID":"dbbc9a37-7489-4812-9f73-43fae8824d1d","Type":"ContainerStarted","Data":"a0b078d1a4a9b93c9426f156903e3273ea9cf777ca99dea992ba3a032c3da27c"} Jul 13 14:53:21.015267 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.015242 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bvxdz_56e9f752-fd9a-4f05-8c86-fb30760212c3/node-ca/0.log" Jul 13 14:53:21.265539 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.265513 2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:21.265692 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.265569 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:21.265939 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.265926 2554 scope.go:117] "RemoveContainer" containerID="436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25" Jul 13 14:53:21.266093 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:21.266075 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mbl9k_openshift-console-operator(c8b6e5d5-b197-4dfc-a471-496cbd3d24ee)\"" pod="openshift-console-operator/console-operator-7866497494-mbl9k" podUID="c8b6e5d5-b197-4dfc-a471-496cbd3d24ee" Jul 13 14:53:21.489263 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.489224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:21.604621 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.604527 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:21.619636 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.616930 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zjf24_7d2dec88-ccda-4b33-85f7-870aece5497b/serve-healthcheck-canary/0.log" Jul 13 14:53:21.746762 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.746726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:21.746953 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.746800 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:21.746953 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:21.746942 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:53:21.747066 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:21.746955 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:53:21.747066 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:21.747022 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:25.747002712 +0000 UTC m=+68.637677796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : configmap "kube-rbac-proxy" not found Jul 13 14:53:21.747066 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:21.747042 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:25.747033034 +0000 UTC m=+68.637708116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : secret "insights-runtime-extractor-tls" not found Jul 13 14:53:21.819328 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.819295 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-w6k66_d746f33c-9114-40c8-b256-1bb5d0d36485/migrator/0.log" Jul 13 14:53:21.829766 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.829734 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:21.966100 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:21.965981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:22.021707 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.021676 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-w6k66_d746f33c-9114-40c8-b256-1bb5d0d36485/graceful-termination/0.log" Jul 13 14:53:22.128170 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.128135 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:22.221956 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.221870 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-q6bnd_bc3a4046-3553-4864-a26c-59f35f1f0da9/kube-storage-version-migrator-operator/0.log" Jul 13 14:53:22.225507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.225466 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:22.397755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.397719 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:22.507314 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:22.507231 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:23.511359 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.511326 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:23.666244 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.666206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:53:23.666405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.666259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:53:23.670530 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.670510 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:53:23.671685 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.671670 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:53:23.676471 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:23.676454 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 14:53:23.676587 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:23.676505 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs podName:213edeac-f8f8-4ffc-864c-829a72c409f3 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:27.676490694 +0000 UTC m=+130.567165775 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs") pod "network-metrics-daemon-vrlkt" (UID: "213edeac-f8f8-4ffc-864c-829a72c409f3") : secret "metrics-daemon-secret" not found Jul 13 14:53:23.678463 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.678439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b09ff867-1f8f-40fc-883f-7afd37b3b88b-original-pull-secret\") pod \"global-pull-secret-syncer-6mfkf\" (UID: \"b09ff867-1f8f-40fc-883f-7afd37b3b88b\") " pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:53:23.767162 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.767072 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:53:23.769463 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.769436 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqw7\" (UniqueName: \"kubernetes.io/projected/e780baeb-3c9c-4b4a-9c31-62e14d730b3a-kube-api-access-pzqw7\") pod \"network-check-target-g2q2r\" (UID: \"e780baeb-3c9c-4b4a-9c31-62e14d730b3a\") " pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:53:23.786159 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.786140 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hw4cv\"" Jul 13 14:53:23.788259 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.788235 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mfkf" Jul 13 14:53:23.793573 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.793555 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:53:23.930978 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.930923 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-4qt4p" podStartSLOduration=3.930901691 podStartE2EDuration="3.930901691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:20.976106285 +0000 UTC m=+63.866781386" watchObservedRunningTime="2026-07-13 14:53:23.930901691 +0000 UTC m=+66.821576794" Jul 13 14:53:23.932460 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.932430 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6mfkf"] Jul 13 14:53:23.936013 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:23.935977 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ff867_1f8f_40fc_883f_7afd37b3b88b.slice/crio-f49200df33ba96349693e0a62ce5568f802f1cc2a1e37e959639c0625ab8e897 WatchSource:0}: Error finding container f49200df33ba96349693e0a62ce5568f802f1cc2a1e37e959639c0625ab8e897: Status 404 returned error can't find the container with id f49200df33ba96349693e0a62ce5568f802f1cc2a1e37e959639c0625ab8e897 Jul 13 14:53:23.943930 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.943906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:23.950151 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.950134 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2q2r"] Jul 13 14:53:23.952447 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:23.952428 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode780baeb_3c9c_4b4a_9c31_62e14d730b3a.slice/crio-761c89eb49c558f6bb41826ca25cbec9aee4d49f3cff642a5e24e9123ca4bb6e WatchSource:0}: Error finding container 761c89eb49c558f6bb41826ca25cbec9aee4d49f3cff642a5e24e9123ca4bb6e: Status 404 returned error can't find the container with id 761c89eb49c558f6bb41826ca25cbec9aee4d49f3cff642a5e24e9123ca4bb6e Jul 13 14:53:23.955292 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:23.955268 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6mfkf" event={"ID":"b09ff867-1f8f-40fc-883f-7afd37b3b88b","Type":"ContainerStarted","Data":"f49200df33ba96349693e0a62ce5568f802f1cc2a1e37e959639c0625ab8e897"} Jul 13 14:53:24.120593 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.120500 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.301527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.301489 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.513856 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.513824 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.667727 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.667685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.801295 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.801213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.961292 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.961257 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2q2r" event={"ID":"e780baeb-3c9c-4b4a-9c31-62e14d730b3a","Type":"ContainerStarted","Data":"a24e087c0da31b87d8211e83ffcfc3bb4033632292ce56d01a9f3c24d84fdb06"} Jul 13 14:53:24.961446 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.961312 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2q2r" event={"ID":"e780baeb-3c9c-4b4a-9c31-62e14d730b3a","Type":"ContainerStarted","Data":"761c89eb49c558f6bb41826ca25cbec9aee4d49f3cff642a5e24e9123ca4bb6e"} Jul 13 14:53:24.961446 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:24.961377 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:53:25.319995 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.319950 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:25.489724 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.489687 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:25.696723 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.696639 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:25.786537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.786508 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:25.786719 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.786571 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:25.786719 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:25.786683 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:53:25.786719 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:25.786703 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:53:25.786864 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:25.786752 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:33.786736002 +0000 UTC m=+76.677411082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : secret "insights-runtime-extractor-tls" not found Jul 13 14:53:25.786864 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:25.786772 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm podName:0ab3b13b-61b0-4ab3-a738-6173ab8b4749 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:33.786762571 +0000 UTC m=+76.677437651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9svjz" (UID: "0ab3b13b-61b0-4ab3-a738-6173ab8b4749") : configmap "kube-rbac-proxy" not found Jul 13 14:53:25.888589 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:25.888556 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.088968 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.088929 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.290895 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.290860 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:26.293747 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.293721 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"image-registry-555977f595-c5jtk\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:26.298966 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.298925 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.399428 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.399345 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.514962 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.514931 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-77ztl\"" Jul 13 14:53:26.522471 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.522437 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:26.603471 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.595583 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:26.603471 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:26.595744 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:26.603471 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:26.596126 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls podName:190cd310-f8ad-4148-85d7-cd456b1d1552 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:42.596075979 +0000 UTC m=+85.486751073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-pb9f5" (UID: "190cd310-f8ad-4148-85d7-cd456b1d1552") : secret "cluster-monitoring-operator-tls" not found Jul 13 14:53:26.686464 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.686403 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.697881 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.697840 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:26.698268 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.697919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:26.698268 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.697949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:26.698268 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.698025 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:26.698433 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:26.698066 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 14:53:26.699844 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:26.698084 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle podName:254d637f-eb4a-4b2e-ac6b-9ac05474709b nodeName:}" failed. No retries permitted until 2026-07-13 14:53:42.698063648 +0000 UTC m=+85.588738740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle") pod "router-default-66644b4cfd-4t6dc" (UID: "254d637f-eb4a-4b2e-ac6b-9ac05474709b") : configmap references non-existent config key: service-ca.crt Jul 13 14:53:26.700633 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:26.700141 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert podName:e632f3ac-8807-4f49-b083-2b41f69e6890 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:42.700103304 +0000 UTC m=+85.590778391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-8km9s" (UID: "e632f3ac-8807-4f49-b083-2b41f69e6890") : secret "networking-console-plugin-cert" not found Jul 13 14:53:26.704197 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.704156 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254d637f-eb4a-4b2e-ac6b-9ac05474709b-metrics-certs\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:26.711524 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.711441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1f2e019-aaa0-4042-99df-69b9fd53c892-metrics-tls\") pod \"dns-default-c9s7d\" (UID: \"d1f2e019-aaa0-4042-99df-69b9fd53c892\") " pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:26.718283 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.718242 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g2q2r" podStartSLOduration=67.718223362 podStartE2EDuration="1m7.718223362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:24.986598514 +0000 UTC m=+67.877273617" watchObservedRunningTime="2026-07-13 14:53:26.718223362 +0000 UTC m=+69.608898465" Jul 13 14:53:26.718631 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.718611 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:53:26.722917 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:26.722889 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eb1b86_a88b_4406_ab8d_d50431ce4cce.slice/crio-399496ed46bd4f29ca8facf88bea4421ee68dbbe47e83cbe1716275778540dd8 WatchSource:0}: Error finding container 399496ed46bd4f29ca8facf88bea4421ee68dbbe47e83cbe1716275778540dd8: Status 404 returned error can't find the container with id 399496ed46bd4f29ca8facf88bea4421ee68dbbe47e83cbe1716275778540dd8 Jul 13 14:53:26.800898 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.800852 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:26.803635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.803609 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-s9jxj\" (UID: \"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:26.853798 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.853769 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mgh5\"" Jul 13 14:53:26.858503 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.858483 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2962n\"" Jul 13 14:53:26.861402 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.861381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:26.866668 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.866649 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" Jul 13 14:53:26.981202 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.981165 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555977f595-c5jtk" event={"ID":"32eb1b86-a88b-4406-ab8d-d50431ce4cce","Type":"ContainerStarted","Data":"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1"} Jul 13 14:53:26.981334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.981232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555977f595-c5jtk" event={"ID":"32eb1b86-a88b-4406-ab8d-d50431ce4cce","Type":"ContainerStarted","Data":"399496ed46bd4f29ca8facf88bea4421ee68dbbe47e83cbe1716275778540dd8"} Jul 13 14:53:26.982543 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:26.982108 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:27.052160 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.050642 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-555977f595-c5jtk" podStartSLOduration=45.050616535 podStartE2EDuration="45.050616535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:27.024666669 +0000 UTC m=+69.915341772" watchObservedRunningTime="2026-07-13 14:53:27.050616535 +0000 UTC m=+69.941291638" Jul 13 14:53:27.052160 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.051903 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c9s7d"] Jul 13 14:53:27.115197 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.115170 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj"] Jul 13 14:53:27.124889 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.124740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:27.494114 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.494030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:27.881930 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.881891 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:27.986474 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.986441 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9s7d" event={"ID":"d1f2e019-aaa0-4042-99df-69b9fd53c892","Type":"ContainerStarted","Data":"cbe38fb15062ec6971cdd157bb2f1c50eb8e6c126d513de43faa8122fa19b39c"} Jul 13 14:53:27.987750 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:27.987724 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" event={"ID":"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca","Type":"ContainerStarted","Data":"3429aa7dde1a042024f60c490dac4e71f73cfe8e1b9dcd31a191b970192c55d3"} Jul 13 14:53:28.103742 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:28.103706 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:28.635061 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:28.635019 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:28.701486 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:28.701436 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:28.881260 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:28.881226 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:29.093305 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:29.093266 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:29.290836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:29.290771 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:29.998363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:29.998325 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6mfkf" event={"ID":"b09ff867-1f8f-40fc-883f-7afd37b3b88b","Type":"ContainerStarted","Data":"f3c3de30994a6ee940ac6be0a804e34a0bb7f47c4bdb4022dd07dc41523779c6"} Jul 13 14:53:30.016327 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:30.016265 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6mfkf" podStartSLOduration=65.675896616 podStartE2EDuration="1m11.016246292s" totalImagesPullingTime="5.340349676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:30.015797723 +0000 UTC m=+72.906472825" watchObservedRunningTime="2026-07-13 14:53:30.016246292 +0000 UTC m=+72.906921393" Jul 13 14:53:30.295131 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:30.295087 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:30.679772 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:30.679640 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:30.781463 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:30.781401 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:30.953977 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:30.953945 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:31.005238 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.004712 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" event={"ID":"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca","Type":"ContainerStarted","Data":"21398d7fe4aba488dae4ef7d4ea4d9f00252130107e007eeece49a3939a03084"} Jul 13 14:53:31.005238 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.004777 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" event={"ID":"a7b8b8c9-b024-4cb9-8e85-ee98e0d5a3ca","Type":"ContainerStarted","Data":"89b22e0a2809edb79402f39c46f7dae121bc5f4fa8379ec2b3f143674978038f"} Jul 13 14:53:31.007940 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.007895 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9s7d" event={"ID":"d1f2e019-aaa0-4042-99df-69b9fd53c892","Type":"ContainerStarted","Data":"c1eac3e5a7ca733bd6d64bb26675771838423cab418b72cbd7c801c7bb2e3ade"} Jul 13 14:53:31.025355 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.025302 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-s9jxj" podStartSLOduration=17.338484013 podStartE2EDuration="21.025280218s" totalImagesPullingTime="3.686796205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:31.023608845 +0000 UTC m=+73.914283940" watchObservedRunningTime="2026-07-13 14:53:31.025280218 +0000 UTC m=+73.915955321" Jul 13 14:53:31.154698 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.154657 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:31.290436 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:31.290403 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.012256 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.012224 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c9s7d" event={"ID":"d1f2e019-aaa0-4042-99df-69b9fd53c892","Type":"ContainerStarted","Data":"036d683022caa052a81618343b5225e0fcff4fc72b4ca91ff7773656db399bc5"} Jul 13 14:53:32.033189 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.033145 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c9s7d" podStartSLOduration=34.234116262 podStartE2EDuration="38.033130229s" totalImagesPullingTime="3.799013967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:32.031484473 +0000 UTC m=+74.922159586" watchObservedRunningTime="2026-07-13 14:53:32.033130229 +0000 UTC m=+74.923805308" Jul 13 14:53:32.091278 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.091244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.326534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.326431 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.400502 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.400468 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.487680 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:32.487645 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:33.019171 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.017638 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:33.869055 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.869007 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:33.869055 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.869069 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:33.869652 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.869625 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:33.871360 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.871342 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ab3b13b-61b0-4ab3-a738-6173ab8b4749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9svjz\" (UID: \"0ab3b13b-61b0-4ab3-a738-6173ab8b4749\") " pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:33.938963 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:33.938930 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9svjz" Jul 13 14:53:34.069385 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:34.069255 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9svjz"] Jul 13 14:53:34.072122 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:34.072097 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab3b13b_61b0_4ab3_a738_6173ab8b4749.slice/crio-ce3a7caa10061c43014dcc2fdc944196c145c8f1283ea217bc0ecee1a3e1846e WatchSource:0}: Error finding container ce3a7caa10061c43014dcc2fdc944196c145c8f1283ea217bc0ecee1a3e1846e: Status 404 returned error can't find the container with id ce3a7caa10061c43014dcc2fdc944196c145c8f1283ea217bc0ecee1a3e1846e Jul 13 14:53:35.026345 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:35.026291 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9svjz" event={"ID":"0ab3b13b-61b0-4ab3-a738-6173ab8b4749","Type":"ContainerStarted","Data":"ce3a7caa10061c43014dcc2fdc944196c145c8f1283ea217bc0ecee1a3e1846e"} Jul 13 14:53:36.275004 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:36.274972 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:36.347904 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:36.347836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:36.673284 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:36.673205 2554 scope.go:117] "RemoveContainer" containerID="436c2fd2e42599f06cb18d842501782d2f12f56c6af15e5cbef15753b4ffeb25" Jul 13 14:53:38.039424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.039395 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 14:53:38.039869 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.039516 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mbl9k" event={"ID":"c8b6e5d5-b197-4dfc-a471-496cbd3d24ee","Type":"ContainerStarted","Data":"35072af29f3b73d1f3bf5f024a428861ef399e7ee0ac820afa3ee88a08fbf4c4"} Jul 13 14:53:38.040378 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.040357 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:38.041473 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.041448 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9svjz" event={"ID":"0ab3b13b-61b0-4ab3-a738-6173ab8b4749","Type":"ContainerStarted","Data":"7a8ced1ab427db3430b64b39ae3e9b8302c632e24961e288b9d5ae0a53ce11fc"} Jul 13 14:53:38.056867 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.056844 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-mbl9k" Jul 13 14:53:38.060806 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.058001 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-mbl9k" podStartSLOduration=23.06980394 podStartE2EDuration="28.05798437s" totalImagesPullingTime="4.98818043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:38.057532489 +0000 UTC m=+80.948207592" watchObservedRunningTime="2026-07-13 14:53:38.05798437 +0000 UTC m=+80.948659475" Jul 13 14:53:38.248260 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.248221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.348147 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.348116 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.448888 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.448860 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.550279 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.550244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.922575 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:38.922519 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:39.023254 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:39.023217 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:39.048357 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:39.048324 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9svjz" event={"ID":"0ab3b13b-61b0-4ab3-a738-6173ab8b4749","Type":"ContainerStarted","Data":"404f850979864f1382ae3ecc4f2ccd0c5e672865cfff8d865f0ff5cd6bd1574f"} Jul 13 14:53:39.982358 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:39.982319 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:40.022653 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.022623 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c9s7d" Jul 13 14:53:40.188103 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.187979 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:40.279006 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.278962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:40.371566 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.371524 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:40.647094 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.647015 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:40.951060 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:40.950977 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:41.055993 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:41.055949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:41.058511 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:41.058488 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9svjz" event={"ID":"0ab3b13b-61b0-4ab3-a738-6173ab8b4749","Type":"ContainerStarted","Data":"6d1df3ff17b444b01ac420dbe9fd475f5d09a53af9e3781fadc8d3067595a362"} Jul 13 14:53:41.079206 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:41.079164 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9svjz" podStartSLOduration=16.490617397 podStartE2EDuration="23.079148802s" totalImagesPullingTime="6.588531405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:53:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:41.077674598 +0000 UTC m=+83.968349701" watchObservedRunningTime="2026-07-13 14:53:41.079148802 +0000 UTC m=+83.969823904" Jul 13 14:53:42.655902 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.655868 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:42.658283 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.658261 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/190cd310-f8ad-4148-85d7-cd456b1d1552-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-pb9f5\" (UID: \"190cd310-f8ad-4148-85d7-cd456b1d1552\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:42.756829 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.756765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:42.757028 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.756885 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:42.757470 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.757451 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/254d637f-eb4a-4b2e-ac6b-9ac05474709b-service-ca-bundle\") pod \"router-default-66644b4cfd-4t6dc\" (UID: \"254d637f-eb4a-4b2e-ac6b-9ac05474709b\") " pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:42.759207 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.759186 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e632f3ac-8807-4f49-b083-2b41f69e6890-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-8km9s\" (UID: \"e632f3ac-8807-4f49-b083-2b41f69e6890\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:42.877182 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.877146 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qgdhk\"" Jul 13 14:53:42.884713 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:42.884688 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" Jul 13 14:53:43.007707 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.007678 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ktnj5\"" Jul 13 14:53:43.015121 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.015097 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" Jul 13 14:53:43.015558 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.015539 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5"] Jul 13 14:53:43.018648 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:43.018621 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190cd310_f8ad_4148_85d7_cd456b1d1552.slice/crio-315fc1ca80588c6b7424a1cfa1be923314ca617ee2ad06047b90464f24e135cf WatchSource:0}: Error finding container 315fc1ca80588c6b7424a1cfa1be923314ca617ee2ad06047b90464f24e135cf: Status 404 returned error can't find the container with id 315fc1ca80588c6b7424a1cfa1be923314ca617ee2ad06047b90464f24e135cf Jul 13 14:53:43.020229 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.020146 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p27m8\"" Jul 13 14:53:43.028030 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.028012 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:43.072166 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.072138 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" event={"ID":"190cd310-f8ad-4148-85d7-cd456b1d1552","Type":"ContainerStarted","Data":"315fc1ca80588c6b7424a1cfa1be923314ca617ee2ad06047b90464f24e135cf"} Jul 13 14:53:43.149427 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.149390 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-8km9s"] Jul 13 14:53:43.152834 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:43.152808 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode632f3ac_8807_4f49_b083_2b41f69e6890.slice/crio-acbd83083fdf69c5df80cfd6027c19970afb904644c66af6378253df7f0f98e3 WatchSource:0}: Error finding container acbd83083fdf69c5df80cfd6027c19970afb904644c66af6378253df7f0f98e3: Status 404 returned error can't find the container with id acbd83083fdf69c5df80cfd6027c19970afb904644c66af6378253df7f0f98e3 Jul 13 14:53:43.171738 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.171716 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66644b4cfd-4t6dc"] Jul 13 14:53:43.174248 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:43.174223 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254d637f_eb4a_4b2e_ac6b_9ac05474709b.slice/crio-ad879df78f3395b678941d966e84119a4463bf29c07d781c09ab180c56f63118 WatchSource:0}: Error finding container ad879df78f3395b678941d966e84119a4463bf29c07d781c09ab180c56f63118: Status 404 returned error can't find the container with id ad879df78f3395b678941d966e84119a4463bf29c07d781c09ab180c56f63118 Jul 13 14:53:43.847655 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.847610 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n"] Jul 13 14:53:43.851117 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.851091 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb"] Jul 13 14:53:43.851273 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.851218 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.855722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.854704 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz"] Jul 13 14:53:43.855722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.854815 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.855722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.855060 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 14:53:43.855722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.855310 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 14:53:43.855722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.855573 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 14:53:43.856456 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.856381 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-79d7f\"" Jul 13 14:53:43.856456 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.856461 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 14:53:43.857329 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.857266 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 13 14:53:43.857732 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.857713 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 13 14:53:43.857874 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.857805 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 13 14:53:43.857874 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.857867 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 13 14:53:43.858554 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.858535 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.862634 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.862605 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz"] Jul 13 14:53:43.862741 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.862649 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 13 14:53:43.868753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868632 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.868753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868670 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.868753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzp2\" (UniqueName: \"kubernetes.io/projected/fb46015c-a55a-4c0a-974d-54c063a41b12-kube-api-access-cmzp2\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.868753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868753 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzjw\" (UniqueName: \"kubernetes.io/projected/6ed91217-7a51-4878-9b9e-1f124304de65-kube-api-access-fzzjw\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868822 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/080987b1-aa5a-4fa7-869b-fa48c36bf08f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868854 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5pm\" (UniqueName: \"kubernetes.io/projected/080987b1-aa5a-4fa7-869b-fa48c36bf08f-kube-api-access-jb5pm\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868900 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ed91217-7a51-4878-9b9e-1f124304de65-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868927 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed91217-7a51-4878-9b9e-1f124304de65-tmp\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.868956 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.869021 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.869007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb46015c-a55a-4c0a-974d-54c063a41b12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.869314 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.869040 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.869884 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.869864 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n"] Jul 13 14:53:43.897710 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.896325 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb"] Jul 13 14:53:43.970111 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970083 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/080987b1-aa5a-4fa7-869b-fa48c36bf08f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970123 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5pm\" (UniqueName: \"kubernetes.io/projected/080987b1-aa5a-4fa7-869b-fa48c36bf08f-kube-api-access-jb5pm\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970170 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ed91217-7a51-4878-9b9e-1f124304de65-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.970252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed91217-7a51-4878-9b9e-1f124304de65-tmp\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.970252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970272 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb46015c-a55a-4c0a-974d-54c063a41b12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970300 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970341 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970366 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzp2\" (UniqueName: \"kubernetes.io/projected/fb46015c-a55a-4c0a-974d-54c063a41b12-kube-api-access-cmzp2\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.970462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970444 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzjw\" (UniqueName: \"kubernetes.io/projected/6ed91217-7a51-4878-9b9e-1f124304de65-kube-api-access-fzzjw\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.970903 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.970877 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/080987b1-aa5a-4fa7-869b-fa48c36bf08f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.971053 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.971034 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed91217-7a51-4878-9b9e-1f124304de65-tmp\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.973957 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.973906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb46015c-a55a-4c0a-974d-54c063a41b12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:43.974854 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.974210 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6ed91217-7a51-4878-9b9e-1f124304de65-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:43.974854 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.974814 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.975309 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.975288 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.975691 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.975669 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-ca\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.976347 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.976305 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/080987b1-aa5a-4fa7-869b-fa48c36bf08f-hub\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.980250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.980218 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-6nd4v"] Jul 13 14:53:43.985147 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.985117 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:53:43.988478 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.988311 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 14:53:43.994079 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.994058 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sr9pf\"" Jul 13 14:53:43.996629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.996611 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5pm\" (UniqueName: \"kubernetes.io/projected/080987b1-aa5a-4fa7-869b-fa48c36bf08f-kube-api-access-jb5pm\") pod \"cluster-proxy-proxy-agent-5f7746ff6b-k8vfb\" (UID: \"080987b1-aa5a-4fa7-869b-fa48c36bf08f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:43.999272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:43.999254 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzp2\" (UniqueName: \"kubernetes.io/projected/fb46015c-a55a-4c0a-974d-54c063a41b12-kube-api-access-cmzp2\") pod \"managed-serviceaccount-addon-agent-6bb597c954-92j2n\" (UID: \"fb46015c-a55a-4c0a-974d-54c063a41b12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:44.016409 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.015138 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 14:53:44.017497 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.017474 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6nd4v"] Jul 13 14:53:44.019815 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.019767 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzjw\" (UniqueName: \"kubernetes.io/projected/6ed91217-7a51-4878-9b9e-1f124304de65-kube-api-access-fzzjw\") pod \"klusterlet-addon-workmgr-5cb4d4dfd-wkpxz\" (UID: \"6ed91217-7a51-4878-9b9e-1f124304de65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:44.059893 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.059836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:44.072005 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.071965 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwfs\" (UniqueName: \"kubernetes.io/projected/6c4e880c-1abc-498e-9179-3aa3fe532d2b-kube-api-access-cwwfs\") pod \"downloads-68c9cbb497-6nd4v\" (UID: \"6c4e880c-1abc-498e-9179-3aa3fe532d2b\") " pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:53:44.078273 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.078242 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" event={"ID":"254d637f-eb4a-4b2e-ac6b-9ac05474709b","Type":"ContainerStarted","Data":"2068e5cfb80df417ca040153cf67c8f332831b0885a50767dfa93e76a56d66ce"} Jul 13 14:53:44.078421 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.078297 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" event={"ID":"254d637f-eb4a-4b2e-ac6b-9ac05474709b","Type":"ContainerStarted","Data":"ad879df78f3395b678941d966e84119a4463bf29c07d781c09ab180c56f63118"} Jul 13 14:53:44.080145 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.080121 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" event={"ID":"e632f3ac-8807-4f49-b083-2b41f69e6890","Type":"ContainerStarted","Data":"acbd83083fdf69c5df80cfd6027c19970afb904644c66af6378253df7f0f98e3"} Jul 13 14:53:44.173424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.173349 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwfs\" (UniqueName: \"kubernetes.io/projected/6c4e880c-1abc-498e-9179-3aa3fe532d2b-kube-api-access-cwwfs\") pod \"downloads-68c9cbb497-6nd4v\" (UID: \"6c4e880c-1abc-498e-9179-3aa3fe532d2b\") " pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:53:44.176214 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.176192 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" Jul 13 14:53:44.185041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.185020 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwfs\" (UniqueName: \"kubernetes.io/projected/6c4e880c-1abc-498e-9179-3aa3fe532d2b-kube-api-access-cwwfs\") pod \"downloads-68c9cbb497-6nd4v\" (UID: \"6c4e880c-1abc-498e-9179-3aa3fe532d2b\") " pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:53:44.190700 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.190682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" Jul 13 14:53:44.198629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.198484 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:44.215049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.215021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:44.319221 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.319180 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:53:44.319604 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.319566 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:44.505265 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.505048 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" podStartSLOduration=34.505023789 podStartE2EDuration="34.505023789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:44.115076517 +0000 UTC m=+87.005751620" watchObservedRunningTime="2026-07-13 14:53:44.505023789 +0000 UTC m=+87.395698891" Jul 13 14:53:44.505481 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.505443 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb"] Jul 13 14:53:44.510205 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:44.510180 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod080987b1_aa5a_4fa7_869b_fa48c36bf08f.slice/crio-8596e24adfbec3e7c854ae1f718f3773678c5a2147387db628acb3f202ee497f WatchSource:0}: Error finding container 8596e24adfbec3e7c854ae1f718f3773678c5a2147387db628acb3f202ee497f: Status 404 returned error can't find the container with id 8596e24adfbec3e7c854ae1f718f3773678c5a2147387db628acb3f202ee497f Jul 13 14:53:44.519467 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.519401 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz"] Jul 13 14:53:44.523130 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:44.523101 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed91217_7a51_4878_9b9e_1f124304de65.slice/crio-92100a7bfc28230de9636b1d09a6d494ef77a7f6664e4ae033c11c03326bda88 WatchSource:0}: Error finding container 92100a7bfc28230de9636b1d09a6d494ef77a7f6664e4ae033c11c03326bda88: Status 404 returned error can't find the container with id 92100a7bfc28230de9636b1d09a6d494ef77a7f6664e4ae033c11c03326bda88 Jul 13 14:53:44.544412 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.544375 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n"] Jul 13 14:53:44.547329 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:44.547283 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-6nd4v"] Jul 13 14:53:44.547636 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:44.547607 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb46015c_a55a_4c0a_974d_54c063a41b12.slice/crio-26b04e80167ad056863434401dc51d4e8560e84dbf340f92c021159e7ec5c2c7 WatchSource:0}: Error finding container 26b04e80167ad056863434401dc51d4e8560e84dbf340f92c021159e7ec5c2c7: Status 404 returned error can't find the container with id 26b04e80167ad056863434401dc51d4e8560e84dbf340f92c021159e7ec5c2c7 Jul 13 14:53:44.550147 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:44.550067 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4e880c_1abc_498e_9179_3aa3fe532d2b.slice/crio-c1ea470f4bf6a186efc5b8e45a8fc558a455dead50c56213f24bb43a34363bd6 WatchSource:0}: Error finding container c1ea470f4bf6a186efc5b8e45a8fc558a455dead50c56213f24bb43a34363bd6: Status 404 returned error can't find the container with id c1ea470f4bf6a186efc5b8e45a8fc558a455dead50c56213f24bb43a34363bd6 Jul 13 14:53:45.028760 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.028681 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:45.031980 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.031954 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:45.085703 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.085674 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" event={"ID":"080987b1-aa5a-4fa7-869b-fa48c36bf08f","Type":"ContainerStarted","Data":"8596e24adfbec3e7c854ae1f718f3773678c5a2147387db628acb3f202ee497f"} Jul 13 14:53:45.086841 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.086815 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6nd4v" event={"ID":"6c4e880c-1abc-498e-9179-3aa3fe532d2b","Type":"ContainerStarted","Data":"c1ea470f4bf6a186efc5b8e45a8fc558a455dead50c56213f24bb43a34363bd6"} Jul 13 14:53:45.088331 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.088272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" event={"ID":"e632f3ac-8807-4f49-b083-2b41f69e6890","Type":"ContainerStarted","Data":"e7f3a11fd4da2877776b3cb2c81af79155031cf6160fec05d75fcff92e5a0e52"} Jul 13 14:53:45.089418 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.089398 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" event={"ID":"fb46015c-a55a-4c0a-974d-54c063a41b12","Type":"ContainerStarted","Data":"26b04e80167ad056863434401dc51d4e8560e84dbf340f92c021159e7ec5c2c7"} Jul 13 14:53:45.090550 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.090531 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" event={"ID":"6ed91217-7a51-4878-9b9e-1f124304de65","Type":"ContainerStarted","Data":"92100a7bfc28230de9636b1d09a6d494ef77a7f6664e4ae033c11c03326bda88"} Jul 13 14:53:45.090831 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.090813 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:45.091867 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.091850 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66644b4cfd-4t6dc" Jul 13 14:53:45.113993 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.113939 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-8km9s" podStartSLOduration=33.897117261 podStartE2EDuration="35.113921391s" totalImagesPullingTime="1.21680413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:45.111889973 +0000 UTC m=+88.002565076" watchObservedRunningTime="2026-07-13 14:53:45.113921391 +0000 UTC m=+88.004596494" Jul 13 14:53:45.267148 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.267095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.362367 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.362230 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.506687 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.506599 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.605935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.605902 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.709245 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.707890 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9"] Jul 13 14:53:45.711753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.711701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.725683 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.725370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:45.729295 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.729272 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9"] Jul 13 14:53:45.729509 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.729491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 14:53:45.729798 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.729764 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gwrkn\"" Jul 13 14:53:45.788213 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.788141 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-wjjb9\" (UID: \"e81b452e-5c6f-4220-b702-d7f0801d8ab3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:45.889665 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.889154 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-wjjb9\" (UID: \"e81b452e-5c6f-4220-b702-d7f0801d8ab3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:45.889665 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:45.889374 2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 13 14:53:45.889665 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:53:45.889443 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates podName:e81b452e-5c6f-4220-b702-d7f0801d8ab3 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:46.389425333 +0000 UTC m=+89.280100428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-wjjb9" (UID: "e81b452e-5c6f-4220-b702-d7f0801d8ab3") : secret "prometheus-operator-admission-webhook-tls" not found Jul 13 14:53:45.985741 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:45.985527 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:46.101727 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.101410 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" event={"ID":"190cd310-f8ad-4148-85d7-cd456b1d1552","Type":"ContainerStarted","Data":"bc0fac69042f1a1bb7dcfcc8c56437ccbe3725c0fe79f1e825169039eafa5b92"} Jul 13 14:53:46.125207 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.124424 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-pb9f5" podStartSLOduration=34.001434249 podStartE2EDuration="36.124402771s" totalImagesPullingTime="2.122968522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:46.120608343 +0000 UTC m=+89.011283446" watchObservedRunningTime="2026-07-13 14:53:46.124402771 +0000 UTC m=+89.015077874" Jul 13 14:53:46.381122 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.381070 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:53:46.394569 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.394089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-wjjb9\" (UID: \"e81b452e-5c6f-4220-b702-d7f0801d8ab3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:46.404876 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.404316 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:53:46.404876 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.404430 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.408670 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.408647 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:53:46.408931 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.408902 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:53:46.411077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.409179 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zvmr4\"" Jul 13 14:53:46.411077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.409359 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:53:46.411077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.409605 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:53:46.411077 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.409892 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:53:46.428252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.422145 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e81b452e-5c6f-4220-b702-d7f0801d8ab3-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-wjjb9\" (UID: \"e81b452e-5c6f-4220-b702-d7f0801d8ab3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:46.494674 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494624 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.494935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494694 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.494935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.494935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494798 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtz95\" (UniqueName: \"kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.494935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494833 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.494935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.494866 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.529556 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.529236 2554 patch_prober.go:28] interesting pod/image-registry-555977f595-c5jtk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 13 14:53:46.529556 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.529298 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-555977f595-c5jtk" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.595819 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.595867 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtz95\" (UniqueName: \"kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.595908 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.595937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.595989 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.596040 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.596874 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.597709 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.597486 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.598446 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.598347 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.611401 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.611377 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.613661 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.613619 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.615204 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.615165 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtz95\" (UniqueName: \"kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95\") pod \"console-bd8f6447f-xqbb4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.647322 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.647231 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:46.765695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.765291 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:46.884722 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:46.884640 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9"] Jul 13 14:53:47.034677 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:47.034605 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:53:47.037219 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:47.037186 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0e6e40_d2e0_4b98_8a3b_d029f7b600c4.slice/crio-f8ad2f18a2a153f57131f6dec30212e5066040ccd47b028ea365474f0804d825 WatchSource:0}: Error finding container f8ad2f18a2a153f57131f6dec30212e5066040ccd47b028ea365474f0804d825: Status 404 returned error can't find the container with id f8ad2f18a2a153f57131f6dec30212e5066040ccd47b028ea365474f0804d825 Jul 13 14:53:47.110534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:47.110439 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" event={"ID":"e81b452e-5c6f-4220-b702-d7f0801d8ab3","Type":"ContainerStarted","Data":"1ee1a78d82604b3397053cd39c80d63e095c5835b3787636f7cf6b5a145cf208"} Jul 13 14:53:47.116681 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:47.116648 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd8f6447f-xqbb4" event={"ID":"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4","Type":"ContainerStarted","Data":"f8ad2f18a2a153f57131f6dec30212e5066040ccd47b028ea365474f0804d825"} Jul 13 14:53:48.380002 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.378590 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:53:48.390947 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.390899 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:48.447016 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.446914 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:53:48.447016 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.447010 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.457421 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.457393 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 14:53:48.478351 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.478324 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:48.515299 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515224 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515299 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515271 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515535 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515371 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515535 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515415 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515535 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515465 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqpf\" (UniqueName: \"kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515535 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515494 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.515743 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.515560 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616561 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616593 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616637 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616663 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616708 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqpf\" (UniqueName: \"kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617334 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.616739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.617958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.617932 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.618554 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.618532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.619103 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.619078 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.619294 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.619270 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.622398 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.622376 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.628321 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.628297 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqpf\" (UniqueName: \"kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.639719 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.639614 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert\") pod \"console-67d575c774-m52vj\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:48.700394 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.700361 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:48.757390 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.757221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:48.764632 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:48.764606 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:49.002362 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:49.002281 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:53:49.419806 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:49.419738 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:49.582743 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:49.582708 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:49.776462 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:49.776416 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:50.017712 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:50.017668 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:50.234989 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:50.234901 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:50.336948 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:50.336908 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:53.659600 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:53.659523 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:53:53.834244 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:53:53.834206 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc473f27e_ee6c_4cec_94d2_af68fd9c3905.slice/crio-30be9655497af47860781d1c0d03254ea487cacf274d893439e3e11e524ce6e6 WatchSource:0}: Error finding container 30be9655497af47860781d1c0d03254ea487cacf274d893439e3e11e524ce6e6: Status 404 returned error can't find the container with id 30be9655497af47860781d1c0d03254ea487cacf274d893439e3e11e524ce6e6 Jul 13 14:53:54.026071 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.025770 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.120138 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.119089 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.156778 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.156732 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" event={"ID":"fb46015c-a55a-4c0a-974d-54c063a41b12","Type":"ContainerStarted","Data":"066a41864ff98ab39e764a68c705ec2ae9223ec817572eee6581b11b2e324717"} Jul 13 14:53:54.158944 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.158558 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" event={"ID":"6ed91217-7a51-4878-9b9e-1f124304de65","Type":"ContainerStarted","Data":"ef13d3193bf26f9d25d01a3faca776de6f82854c92f5322bef6e2764d1cef346"} Jul 13 14:53:54.158944 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.158711 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:54.160308 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.160289 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" Jul 13 14:53:54.160406 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.160329 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" event={"ID":"e81b452e-5c6f-4220-b702-d7f0801d8ab3","Type":"ContainerStarted","Data":"db6074284e800fec2eccd44cda29c3c440235c9412a8e67b2fe4d3ac46d64a14"} Jul 13 14:53:54.160572 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.160557 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:54.167066 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.167044 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" Jul 13 14:53:54.169066 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.168723 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d575c774-m52vj" event={"ID":"c473f27e-ee6c-4cec-94d2-af68fd9c3905","Type":"ContainerStarted","Data":"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4"} Jul 13 14:53:54.169066 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.168763 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d575c774-m52vj" event={"ID":"c473f27e-ee6c-4cec-94d2-af68fd9c3905","Type":"ContainerStarted","Data":"30be9655497af47860781d1c0d03254ea487cacf274d893439e3e11e524ce6e6"} Jul 13 14:53:54.172335 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.172314 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" event={"ID":"080987b1-aa5a-4fa7-869b-fa48c36bf08f","Type":"ContainerStarted","Data":"4e423a61d6637cb61c5ba42ecbd28ed6dce1bd8ac8f4831138ce95ed8d3c7956"} Jul 13 14:53:54.172465 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.172447 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" event={"ID":"080987b1-aa5a-4fa7-869b-fa48c36bf08f","Type":"ContainerStarted","Data":"dbc9aeee1fb34af84ba452d6d3cc054f4572f53c17c6ce22cd089c01582cdf60"} Jul 13 14:53:54.174627 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.174601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd8f6447f-xqbb4" event={"ID":"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4","Type":"ContainerStarted","Data":"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca"} Jul 13 14:53:54.191147 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.191101 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb597c954-92j2n" podStartSLOduration=2.234979115 podStartE2EDuration="11.191085261s" totalImagesPullingTime="8.956106146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.190892394 +0000 UTC m=+97.081567495" watchObservedRunningTime="2026-07-13 14:53:54.191085261 +0000 UTC m=+97.081760366" Jul 13 14:53:54.221464 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.221409 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-wjjb9" podStartSLOduration=2.25530576 podStartE2EDuration="9.221388669s" totalImagesPullingTime="6.966082909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.219654662 +0000 UTC m=+97.110329764" watchObservedRunningTime="2026-07-13 14:53:54.221388669 +0000 UTC m=+97.112063752" Jul 13 14:53:54.251946 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.251900 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" podStartSLOduration=1.947522137 podStartE2EDuration="11.251885542s" totalImagesPullingTime="9.304363405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.251015227 +0000 UTC m=+97.141690330" watchObservedRunningTime="2026-07-13 14:53:54.251885542 +0000 UTC m=+97.142560643" Jul 13 14:53:54.280648 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.280596 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bd8f6447f-xqbb4" podStartSLOduration=1.45799796 podStartE2EDuration="8.280575419s" totalImagesPullingTime="6.822577459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.27967024 +0000 UTC m=+97.170345343" watchObservedRunningTime="2026-07-13 14:53:54.280575419 +0000 UTC m=+97.171250525" Jul 13 14:53:54.307010 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.306955 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d575c774-m52vj" podStartSLOduration=6.306934443 podStartE2EDuration="6.306934443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.304881306 +0000 UTC m=+97.195556409" watchObservedRunningTime="2026-07-13 14:53:54.306934443 +0000 UTC m=+97.197609548" Jul 13 14:53:54.341118 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.341038 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.504888 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.504848 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.746139 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.745944 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.878436 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:54.878404 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:55.183363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.182834 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f7746ff6b-k8vfb" event={"ID":"080987b1-aa5a-4fa7-869b-fa48c36bf08f","Type":"ContainerStarted","Data":"0708d26738aae9518e25140dd0d51c97cbe2ac6a8e35067cc8a42c0bb55f42b1"} Jul 13 14:53:55.302552 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.302512 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:55.719866 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.719804 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:55.818467 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.818432 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:55.968743 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.968712 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g2q2r" Jul 13 14:53:55.993262 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:55.993178 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cb4d4dfd-wkpxz" podStartSLOduration=3.686026261 podStartE2EDuration="12.993160374s" totalImagesPullingTime="9.307134113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:54.362723754 +0000 UTC m=+97.253398858" watchObservedRunningTime="2026-07-13 14:53:55.993160374 +0000 UTC m=+98.883835469" Jul 13 14:53:56.766095 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:56.766053 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:56.766281 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:56.766119 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:56.771701 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:56.771678 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:57.203563 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:57.203482 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:53:57.547737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:57.547686 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:58.765712 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:58.765677 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:58.766164 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:58.765749 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:58.771539 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:58.771516 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:59.216485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:59.216402 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:53:59.275554 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:59.275524 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:53:59.598207 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:59.598172 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:59.794442 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:59.794406 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:59.910823 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:53:59.910681 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:00.011848 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:00.011809 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:00.935943 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:00.935884 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.157718 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.157662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.282817 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.281321 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.300867 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.300478 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ndcgp"] Jul 13 14:54:01.414910 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.414834 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.431595 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.431561 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.436836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.436183 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 14:54:01.436836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.436227 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 14:54:01.436836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.436283 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 14:54:01.436836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.436337 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q98rp\"" Jul 13 14:54:01.436836 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.436404 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 14:54:01.543333 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543250 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-textfile\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543333 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543299 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543398 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7xg\" (UniqueName: \"kubernetes.io/projected/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-kube-api-access-rc7xg\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-metrics-client-ca\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543501 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-sys\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-wtmp\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543799 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543570 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543799 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543619 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-root\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.543799 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.543644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.636753 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.636128 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644778 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-sys\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-wtmp\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644868 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644915 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-root\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644971 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-textfile\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.644997 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.645044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7xg\" (UniqueName: \"kubernetes.io/projected/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-kube-api-access-rc7xg\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.645087 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-metrics-client-ca\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.645690 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-metrics-client-ca\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.646820 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.645757 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-sys\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.648600 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:54:01.647570 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 14:54:01.648600 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:54:01.647636 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls podName:06c3f473-a1b4-45e3-87ee-e1c87458f4cf nodeName:}" failed. No retries permitted until 2026-07-13 14:54:02.147616247 +0000 UTC m=+105.038291338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls") pod "node-exporter-ndcgp" (UID: "06c3f473-a1b4-45e3-87ee-e1c87458f4cf") : secret "node-exporter-tls" not found Jul 13 14:54:01.648600 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.648144 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-root\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.648600 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.648547 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-textfile\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.648895 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.648872 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-wtmp\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.649215 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.649057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.650271 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.650250 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.663270 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.663246 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7xg\" (UniqueName: \"kubernetes.io/projected/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-kube-api-access-rc7xg\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:01.739226 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.739104 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.842422 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.842313 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:01.961515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:01.961454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:02.062171 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.062106 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:02.149711 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.149634 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:02.152944 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.152921 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06c3f473-a1b4-45e3-87ee-e1c87458f4cf-node-exporter-tls\") pod \"node-exporter-ndcgp\" (UID: \"06c3f473-a1b4-45e3-87ee-e1c87458f4cf\") " pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:02.162684 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.162653 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:02.263253 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.263213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:02.347102 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.347068 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndcgp" Jul 13 14:54:02.356253 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:02.356220 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:03.338093 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:03.338063 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:03.438168 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:03.438132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:03.820177 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:03.820138 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:03.919498 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:03.919457 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:04.182987 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:04.182949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:04.185970 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:54:04.185937 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c3f473_a1b4_45e3_87ee_e1c87458f4cf.slice/crio-59570def6a0c6013bab6289480259589483fb6f9f84fae7f6c4dcb5ccfae25a0 WatchSource:0}: Error finding container 59570def6a0c6013bab6289480259589483fb6f9f84fae7f6c4dcb5ccfae25a0: Status 404 returned error can't find the container with id 59570def6a0c6013bab6289480259589483fb6f9f84fae7f6c4dcb5ccfae25a0 Jul 13 14:54:04.240238 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:04.240206 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndcgp" event={"ID":"06c3f473-a1b4-45e3-87ee-e1c87458f4cf","Type":"ContainerStarted","Data":"59570def6a0c6013bab6289480259589483fb6f9f84fae7f6c4dcb5ccfae25a0"} Jul 13 14:54:04.256022 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:04.255981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:04.319566 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:04.319531 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:04.963888 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:04.963766 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:05.248493 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.248405 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-6nd4v" event={"ID":"6c4e880c-1abc-498e-9179-3aa3fe532d2b","Type":"ContainerStarted","Data":"1fd809f4e917e622748f54fed4505f8fc7b4b9a235c571c247c2f589737fa0cd"} Jul 13 14:54:05.249385 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.249060 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:54:05.270210 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.270180 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-6nd4v" Jul 13 14:54:05.270681 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.270633 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-6nd4v" podStartSLOduration=2.274505993 podStartE2EDuration="22.270613175s" totalImagesPullingTime="19.996107182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:05.268735985 +0000 UTC m=+108.159411089" watchObservedRunningTime="2026-07-13 14:54:05.270613175 +0000 UTC m=+108.161288277" Jul 13 14:54:05.337052 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.336885 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:05.958294 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:05.958262 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:06.091858 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:06.091821 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:06.255956 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:06.255875 2554 generic.go:356] "Generic (PLEG): container finished" podID="06c3f473-a1b4-45e3-87ee-e1c87458f4cf" containerID="9dca58c7799c531bcf7302445ce07026dc30f81024bf25c34dd20acadc16ec7a" exitCode=0 Jul 13 14:54:06.256124 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:06.255961 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndcgp" event={"ID":"06c3f473-a1b4-45e3-87ee-e1c87458f4cf","Type":"ContainerDied","Data":"9dca58c7799c531bcf7302445ce07026dc30f81024bf25c34dd20acadc16ec7a"} Jul 13 14:54:06.260340 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:06.260312 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:06.355643 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:06.355602 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:07.263686 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:07.263640 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndcgp" event={"ID":"06c3f473-a1b4-45e3-87ee-e1c87458f4cf","Type":"ContainerStarted","Data":"0206cdf44042fbe1ac05bfa683b87ba0accbf292c7af8e2e6b4587b66d4fe7e8"} Jul 13 14:54:07.264159 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:07.263709 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndcgp" event={"ID":"06c3f473-a1b4-45e3-87ee-e1c87458f4cf","Type":"ContainerStarted","Data":"ad1c2c87dd1382b2bfb5d17e2c06103ed70cdde44c90e8f3c68b67fbf82532b9"} Jul 13 14:54:07.287287 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:07.287239 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ndcgp" podStartSLOduration=4.450813617 podStartE2EDuration="6.28722404s" totalImagesPullingTime="1.770689876s" totalInitContainerRuntime="65.720547ms" isStatefulPod=true podCreationTimestamp="2026-07-13 14:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:07.285994165 +0000 UTC m=+110.176669267" watchObservedRunningTime="2026-07-13 14:54:07.28722404 +0000 UTC m=+110.177899164" Jul 13 14:54:07.493519 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:07.493486 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:08.160929 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:08.160896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:08.261355 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:08.261314 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:11.680088 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:11.680055 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:54:11.759558 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:11.759527 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:11.857591 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:11.857550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:11.946681 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:11.946598 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:12.001332 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:12.001294 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:12.828982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:12.828950 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:12.929047 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:12.929011 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:13.395079 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:13.395034 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:14.807270 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:14.807234 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:15.140553 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:15.140468 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:15.200115 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:15.200084 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:21.259809 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:21.259752 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:21.338591 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:21.338556 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:22.327119 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:22.327087 2554 generic.go:356] "Generic (PLEG): container finished" podID="c26e4564-e76c-4b18-81ee-cb396ab0d1c3" containerID="69625a43fcfddbfb30bd540b6292de65cf34f3fef274b3d015039bf6deb56335" exitCode=0 Jul 13 14:54:22.327441 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:22.327159 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" event={"ID":"c26e4564-e76c-4b18-81ee-cb396ab0d1c3","Type":"ContainerDied","Data":"69625a43fcfddbfb30bd540b6292de65cf34f3fef274b3d015039bf6deb56335"} Jul 13 14:54:22.327481 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:22.327445 2554 scope.go:117] "RemoveContainer" containerID="69625a43fcfddbfb30bd540b6292de65cf34f3fef274b3d015039bf6deb56335" Jul 13 14:54:23.333039 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:23.333008 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qbz7z" event={"ID":"c26e4564-e76c-4b18-81ee-cb396ab0d1c3","Type":"ContainerStarted","Data":"5b4b1fa98883a5e61303e0718c5a5aa2073804db9aeaf751e56fad32ebacf42a"} Jul 13 14:54:23.832458 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:23.832428 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:54:23.833407 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:23.832777 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:23.926890 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:23.926856 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:24.062420 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.062380 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:24.119166 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.119069 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:24.301919 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.301855 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bd8f6447f-xqbb4" podUID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" containerName="console" containerID="cri-o://982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca" gracePeriod=15 Jul 13 14:54:24.584563 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.584531 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:54:24.665972 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.665941 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.665972 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.665975 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.666223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.665992 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.666223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666040 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtz95\" (UniqueName: \"kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.666223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666169 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.666223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666220 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config\") pod \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\" (UID: \"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4\") " Jul 13 14:54:24.666573 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666536 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:24.666695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666536 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:24.666695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.666598 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config" (OuterVolumeSpecName: "console-config") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:24.668410 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.668389 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:24.668635 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.668611 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95" (OuterVolumeSpecName: "kube-api-access-wtz95") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "kube-api-access-wtz95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:24.668689 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.668637 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" (UID: "6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:24.767453 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767426 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtz95\" (UniqueName: \"kubernetes.io/projected/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-kube-api-access-wtz95\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:24.767453 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767454 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-oauth-config\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:24.767629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767463 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-config\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:24.767629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767472 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-console-serving-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:24.767629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767480 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-oauth-serving-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:24.767629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:24.767489 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:25.343592 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.343558 2554 generic.go:356] "Generic (PLEG): container finished" podID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" containerID="982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca" exitCode=0 Jul 13 14:54:25.343755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.343603 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd8f6447f-xqbb4" Jul 13 14:54:25.343755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.343618 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd8f6447f-xqbb4" event={"ID":"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4","Type":"ContainerDied","Data":"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca"} Jul 13 14:54:25.343755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.343664 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd8f6447f-xqbb4" event={"ID":"6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4","Type":"ContainerDied","Data":"f8ad2f18a2a153f57131f6dec30212e5066040ccd47b028ea365474f0804d825"} Jul 13 14:54:25.343755 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.343687 2554 scope.go:117] "RemoveContainer" containerID="982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca" Jul 13 14:54:25.352134 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.352118 2554 scope.go:117] "RemoveContainer" containerID="982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca" Jul 13 14:54:25.352386 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:54:25.352366 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca\": container with ID starting with 982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca not found: ID does not exist" containerID="982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca" Jul 13 14:54:25.352431 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.352395 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca"} err="failed to get container status \"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca\": rpc error: code = NotFound desc = could not find container \"982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca\": container with ID starting with 982f9c0fe82111f031db856db097688b1edcb53771ce14d6b6dec2aeee8cd5ca not found: ID does not exist" Jul 13 14:54:25.364265 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.364240 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:54:25.367765 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.367746 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bd8f6447f-xqbb4"] Jul 13 14:54:25.678563 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:25.678485 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" path="/var/lib/kubelet/pods/6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4/volumes" Jul 13 14:54:27.691217 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:27.691190 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:54:27.693424 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:27.693406 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213edeac-f8f8-4ffc-864c-829a72c409f3-metrics-certs\") pod \"network-metrics-daemon-vrlkt\" (UID: \"213edeac-f8f8-4ffc-864c-829a72c409f3\") " pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:54:27.787287 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:27.787258 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bz4r7\"" Jul 13 14:54:27.794899 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:27.794881 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrlkt" Jul 13 14:54:27.946027 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:27.946005 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrlkt"] Jul 13 14:54:27.948629 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:54:27.948601 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213edeac_f8f8_4ffc_864c_829a72c409f3.slice/crio-d9b3ad24b5e0eea5c69f971126245e87ee6ee78cac0c5d7eacbed5ebc49385a6 WatchSource:0}: Error finding container d9b3ad24b5e0eea5c69f971126245e87ee6ee78cac0c5d7eacbed5ebc49385a6: Status 404 returned error can't find the container with id d9b3ad24b5e0eea5c69f971126245e87ee6ee78cac0c5d7eacbed5ebc49385a6 Jul 13 14:54:28.359307 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:28.359275 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrlkt" event={"ID":"213edeac-f8f8-4ffc-864c-829a72c409f3","Type":"ContainerStarted","Data":"d9b3ad24b5e0eea5c69f971126245e87ee6ee78cac0c5d7eacbed5ebc49385a6"} Jul 13 14:54:30.370001 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.369963 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrlkt" event={"ID":"213edeac-f8f8-4ffc-864c-829a72c409f3","Type":"ContainerStarted","Data":"d2401d0a8a1c66a6adbb993dd04032605960475c7bd1bd109511ed0a203565b6"} Jul 13 14:54:30.370475 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.370026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrlkt" event={"ID":"213edeac-f8f8-4ffc-864c-829a72c409f3","Type":"ContainerStarted","Data":"5f209048b82ec7e378590775634298304960c7ac7adae62aee0d41b32dfc1271"} Jul 13 14:54:30.388301 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.388252 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vrlkt" podStartSLOduration=130.897268235 podStartE2EDuration="2m12.388238644s" totalImagesPullingTime="1.490970409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:30.38713431 +0000 UTC m=+133.277809415" watchObservedRunningTime="2026-07-13 14:54:30.388238644 +0000 UTC m=+133.278913788" Jul 13 14:54:30.473504 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.473462 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:30.574938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.574904 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:30.631252 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:30.631167 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:36.254849 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:36.254812 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:36.313583 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:36.313539 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:36.705233 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:36.705195 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-555977f595-c5jtk" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerName="registry" containerID="cri-o://939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1" gracePeriod=30 Jul 13 14:54:37.007388 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.007364 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:54:37.071018 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.070982 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071018 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071027 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v26k\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071267 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071084 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071267 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071234 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071375 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071291 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071375 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071324 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071375 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071361 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071512 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071393 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") pod \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\" (UID: \"32eb1b86-a88b-4406-ab8d-d50431ce4cce\") " Jul 13 14:54:37.071512 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071422 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:37.071710 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071663 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:37.071847 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071801 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-certificates\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.071847 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.071822 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32eb1b86-a88b-4406-ab8d-d50431ce4cce-trusted-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.073873 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.073833 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k" (OuterVolumeSpecName: "kube-api-access-9v26k") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "kube-api-access-9v26k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:37.073987 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.073931 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:37.074071 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.074041 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:37.074127 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.074069 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:37.074177 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.074153 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:37.079910 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.079885 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "32eb1b86-a88b-4406-ab8d-d50431ce4cce" (UID: "32eb1b86-a88b-4406-ab8d-d50431ce4cce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:54:37.172764 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172728 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-image-registry-private-configuration\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.172764 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172758 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32eb1b86-a88b-4406-ab8d-d50431ce4cce-ca-trust-extracted\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.172764 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172770 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-bound-sa-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.173013 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172779 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-registry-tls\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.173013 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172808 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32eb1b86-a88b-4406-ab8d-d50431ce4cce-installation-pull-secrets\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.173013 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.172816 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v26k\" (UniqueName: \"kubernetes.io/projected/32eb1b86-a88b-4406-ab8d-d50431ce4cce-kube-api-access-9v26k\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:37.399628 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.399593 2554 generic.go:356] "Generic (PLEG): container finished" podID="bc3a4046-3553-4864-a26c-59f35f1f0da9" containerID="4302629f027ef4908492ffcd6dd93ea7cb348e275be313e2378a1cd37d258514" exitCode=0 Jul 13 14:54:37.400132 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.399669 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" event={"ID":"bc3a4046-3553-4864-a26c-59f35f1f0da9","Type":"ContainerDied","Data":"4302629f027ef4908492ffcd6dd93ea7cb348e275be313e2378a1cd37d258514"} Jul 13 14:54:37.400132 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.400088 2554 scope.go:117] "RemoveContainer" containerID="4302629f027ef4908492ffcd6dd93ea7cb348e275be313e2378a1cd37d258514" Jul 13 14:54:37.400938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.400914 2554 generic.go:356] "Generic (PLEG): container finished" podID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerID="939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1" exitCode=0 Jul 13 14:54:37.401049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.400951 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555977f595-c5jtk" event={"ID":"32eb1b86-a88b-4406-ab8d-d50431ce4cce","Type":"ContainerDied","Data":"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1"} Jul 13 14:54:37.401049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.400972 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555977f595-c5jtk" Jul 13 14:54:37.401049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.400980 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555977f595-c5jtk" event={"ID":"32eb1b86-a88b-4406-ab8d-d50431ce4cce","Type":"ContainerDied","Data":"399496ed46bd4f29ca8facf88bea4421ee68dbbe47e83cbe1716275778540dd8"} Jul 13 14:54:37.401049 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.401005 2554 scope.go:117] "RemoveContainer" containerID="939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1" Jul 13 14:54:37.410436 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.410421 2554 scope.go:117] "RemoveContainer" containerID="939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1" Jul 13 14:54:37.410723 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:54:37.410706 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1\": container with ID starting with 939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1 not found: ID does not exist" containerID="939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1" Jul 13 14:54:37.410778 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.410730 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1"} err="failed to get container status \"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1\": rpc error: code = NotFound desc = could not find container \"939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1\": container with ID starting with 939b3ce461921802bd115fa31c56fe6791b0f610899ff230081b6ae0dee11fc1 not found: ID does not exist" Jul 13 14:54:37.435567 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.435533 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:54:37.441834 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.441811 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-555977f595-c5jtk"] Jul 13 14:54:37.679098 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:37.679024 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" path="/var/lib/kubelet/pods/32eb1b86-a88b-4406-ab8d-d50431ce4cce/volumes" Jul 13 14:54:38.410773 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:38.410738 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-q6bnd" event={"ID":"bc3a4046-3553-4864-a26c-59f35f1f0da9","Type":"ContainerStarted","Data":"e8098ad0d5e79c887a4ccf10d297237c9d7e5f8efeaf50d50d6890edae679d9c"} Jul 13 14:54:42.428607 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:42.428572 2554 generic.go:356] "Generic (PLEG): container finished" podID="35ca4899-57dd-4122-a6aa-031e069d9f82" containerID="4f575d5fe79130613c4ccced3a7adebe8f83e597b78f678b1122572ceeb60c28" exitCode=0 Jul 13 14:54:42.429106 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:42.428647 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" event={"ID":"35ca4899-57dd-4122-a6aa-031e069d9f82","Type":"ContainerDied","Data":"4f575d5fe79130613c4ccced3a7adebe8f83e597b78f678b1122572ceeb60c28"} Jul 13 14:54:42.429106 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:42.428992 2554 scope.go:117] "RemoveContainer" containerID="4f575d5fe79130613c4ccced3a7adebe8f83e597b78f678b1122572ceeb60c28" Jul 13 14:54:43.435579 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:43.435541 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-j4cn7" event={"ID":"35ca4899-57dd-4122-a6aa-031e069d9f82","Type":"ContainerStarted","Data":"91679a1170673834a17c8ef890c4d1d87475438daa87291efefb1e05ede8a1c4"} Jul 13 14:54:48.854414 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:48.854352 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67d575c774-m52vj" podUID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" containerName="console" containerID="cri-o://476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4" gracePeriod=15 Jul 13 14:54:49.130494 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.130435 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:54:49.279180 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279141 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279206 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279233 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279266 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279300 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279319 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279363 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279340 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbqpf\" (UniqueName: \"kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf\") pod \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\" (UID: \"c473f27e-ee6c-4cec-94d2-af68fd9c3905\") " Jul 13 14:54:49.279705 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279675 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca" (OuterVolumeSpecName: "service-ca") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:49.279760 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279700 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:49.279760 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279736 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config" (OuterVolumeSpecName: "console-config") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:49.279878 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.279860 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:54:49.281466 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.281442 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:49.281466 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.281459 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf" (OuterVolumeSpecName: "kube-api-access-kbqpf") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "kube-api-access-kbqpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:54:49.281591 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.281481 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c473f27e-ee6c-4cec-94d2-af68fd9c3905" (UID: "c473f27e-ee6c-4cec-94d2-af68fd9c3905"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:54:49.380058 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380024 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-oauth-serving-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380058 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380054 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380247 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380066 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-oauth-config\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380247 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380076 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-config\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380247 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380085 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c473f27e-ee6c-4cec-94d2-af68fd9c3905-trusted-ca-bundle\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380247 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380092 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c473f27e-ee6c-4cec-94d2-af68fd9c3905-console-serving-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.380247 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.380100 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqpf\" (UniqueName: \"kubernetes.io/projected/c473f27e-ee6c-4cec-94d2-af68fd9c3905-kube-api-access-kbqpf\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:54:49.462127 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.462043 2554 generic.go:356] "Generic (PLEG): container finished" podID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" containerID="476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4" exitCode=0 Jul 13 14:54:49.462127 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.462097 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d575c774-m52vj" Jul 13 14:54:49.462288 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.462126 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d575c774-m52vj" event={"ID":"c473f27e-ee6c-4cec-94d2-af68fd9c3905","Type":"ContainerDied","Data":"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4"} Jul 13 14:54:49.462288 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.462179 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d575c774-m52vj" event={"ID":"c473f27e-ee6c-4cec-94d2-af68fd9c3905","Type":"ContainerDied","Data":"30be9655497af47860781d1c0d03254ea487cacf274d893439e3e11e524ce6e6"} Jul 13 14:54:49.462288 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.462200 2554 scope.go:117] "RemoveContainer" containerID="476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4" Jul 13 14:54:49.471352 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.471336 2554 scope.go:117] "RemoveContainer" containerID="476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4" Jul 13 14:54:49.471603 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:54:49.471586 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4\": container with ID starting with 476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4 not found: ID does not exist" containerID="476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4" Jul 13 14:54:49.471650 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.471612 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4"} err="failed to get container status \"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4\": rpc error: code = NotFound desc = could not find container \"476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4\": container with ID starting with 476db7b0b1dc89a3106930246498fbf5963805217c147fcd17cfd69403f7c9f4 not found: ID does not exist" Jul 13 14:54:49.487395 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.487372 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:54:49.491040 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.491022 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d575c774-m52vj"] Jul 13 14:54:49.678278 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:54:49.678246 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" path="/var/lib/kubelet/pods/c473f27e-ee6c-4cec-94d2-af68fd9c3905/volumes" Jul 13 14:55:17.346674 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:17.346563 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:17.406756 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:17.406718 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:29.300593 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:29.300550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:29.364169 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:29.364132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:36.275022 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:36.274984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:36.373387 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:36.373353 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:38.544138 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:38.544102 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:38.947507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:38.947424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:49.310173 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:49.310132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:49.412022 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:49.411981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:49.500421 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:55:49.500376 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:13.461902 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:56:13.461870 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:13.523901 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:56:13.523866 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:40.059394 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:56:40.059358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:40.153584 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:56:40.153505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:01.497326 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:01.497283 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:02.889576 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:02.889544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:03.550042 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:03.550005 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:05.013053 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:05.013020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:05.073862 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:05.073830 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:06.746006 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:06.745973 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:06.867582 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:06.867553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:07.056707 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:07.056668 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:07.147023 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:07.146991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:08.818426 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:08.818386 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:09.422835 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:09.422779 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:12.015675 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:12.015641 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:12.115192 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:12.115156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:12.986336 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:12.986279 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:17.528653 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:17.528619 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 14:57:17.530511 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:17.530491 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 14:57:17.534771 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:17.534752 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:57:17.536223 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:17.536202 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 14:57:17.540485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:17.540469 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 14:57:18.260165 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:18.260127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:20.012136 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:20.012096 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:20.107347 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:20.107312 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:20.356263 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:20.356164 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:20.453887 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:20.453857 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:26.340253 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:26.340211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:26.434695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:26.434664 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:28.149767 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:28.149732 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:29.029938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:29.029900 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:29.090671 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:29.090629 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:44.445938 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:44.445902 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.468906 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.468870 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2"] Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469190 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" containerName="console" Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469202 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" containerName="console" Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469222 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerName="registry" Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469227 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerName="registry" Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469233 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" containerName="console" Jul 13 14:57:45.469272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469239 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" containerName="console" Jul 13 14:57:45.469478 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469284 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c473f27e-ee6c-4cec-94d2-af68fd9c3905" containerName="console" Jul 13 14:57:45.469478 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469294 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0e6e40-d2e0-4b98-8a3b-d029f7b600c4" containerName="console" Jul 13 14:57:45.469478 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.469304 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="32eb1b86-a88b-4406-ab8d-d50431ce4cce" containerName="registry" Jul 13 14:57:45.471250 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.471228 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.475387 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.475367 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 14:57:45.475509 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.475428 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 14:57:45.475963 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.475942 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 14:57:45.476074 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.476003 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 14:57:45.477293 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.477258 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qfmz\"" Jul 13 14:57:45.492355 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.492334 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2"] Jul 13 14:57:45.497287 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.497265 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98pw4\" (UniqueName: \"kubernetes.io/projected/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-kube-api-access-98pw4\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.497376 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.497308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.497417 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.497399 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.497450 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.497440 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.497507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.497486 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598313 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98pw4\" (UniqueName: \"kubernetes.io/projected/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-kube-api-access-98pw4\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598492 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598319 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598492 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598492 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598492 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598443 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.598825 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.598771 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.600824 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.600780 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.600977 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.600956 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.600977 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.600969 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.615361 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.615336 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98pw4\" (UniqueName: \"kubernetes.io/projected/3c77deb1-ddf1-465e-b47b-8fd41a709c4e-kube-api-access-98pw4\") pod \"opendatahub-operator-controller-manager-67985f7df8-h5xt2\" (UID: \"3c77deb1-ddf1-465e-b47b-8fd41a709c4e\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.740521 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.740439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.782409 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.782385 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:45.840715 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.840669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.910507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.910463 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.923104 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.923084 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2"] Jul 13 14:57:45.926047 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:57:45.926016 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c77deb1_ddf1_465e_b47b_8fd41a709c4e.slice/crio-57bee1ff461cc6374e54a28fbd69a62cc5b454c4c81ecb623401adc9b5c2760f WatchSource:0}: Error finding container 57bee1ff461cc6374e54a28fbd69a62cc5b454c4c81ecb623401adc9b5c2760f: Status 404 returned error can't find the container with id 57bee1ff461cc6374e54a28fbd69a62cc5b454c4c81ecb623401adc9b5c2760f Jul 13 14:57:45.927748 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.927727 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:57:45.972416 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:45.972386 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:46.076869 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:46.076840 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:46.160675 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:46.160647 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" event={"ID":"3c77deb1-ddf1-465e-b47b-8fd41a709c4e","Type":"ContainerStarted","Data":"57bee1ff461cc6374e54a28fbd69a62cc5b454c4c81ecb623401adc9b5c2760f"} Jul 13 14:57:49.179440 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:49.179358 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" event={"ID":"3c77deb1-ddf1-465e-b47b-8fd41a709c4e","Type":"ContainerStarted","Data":"9a4acda96a84047499ad08f695790aa6fc269f4913b693fe05c5075a01945449"} Jul 13 14:57:50.185111 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:50.185032 2554 generic.go:356] "Generic (PLEG): container finished" podID="3c77deb1-ddf1-465e-b47b-8fd41a709c4e" containerID="9a4acda96a84047499ad08f695790aa6fc269f4913b693fe05c5075a01945449" exitCode=0 Jul 13 14:57:50.185553 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:50.185118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" event={"ID":"3c77deb1-ddf1-465e-b47b-8fd41a709c4e","Type":"ContainerDied","Data":"9a4acda96a84047499ad08f695790aa6fc269f4913b693fe05c5075a01945449"} Jul 13 14:57:51.193276 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:51.193243 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" event={"ID":"3c77deb1-ddf1-465e-b47b-8fd41a709c4e","Type":"ContainerStarted","Data":"d649759b503c488b5e53d015a94453af3c0c6664ab457ac1a8cdf5589fae48be"} Jul 13 14:57:51.193738 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:51.193367 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:57:51.221704 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:51.221651 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" podStartSLOduration=1.949532031 podStartE2EDuration="6.221631417s" totalImagesPullingTime="3.243562116s" totalInitContainerRuntime="1.02853727s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:57:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:57:51.219095378 +0000 UTC m=+334.109770481" watchObservedRunningTime="2026-07-13 14:57:51.221631417 +0000 UTC m=+334.112306521" Jul 13 14:57:51.967515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:51.967476 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:53.918587 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:53.918549 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:53.973581 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:53.973524 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:57.460862 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:57.460822 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:57.554360 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:57.554316 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:59.345639 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:59.345606 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:59.456562 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:57:59.456536 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:02.200955 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:02.200925 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-h5xt2" Jul 13 14:58:04.916260 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:04.916225 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:05.123644 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.123602 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:05.510516 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.510480 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:05.659152 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.659124 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng"] Jul 13 14:58:05.670800 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.670761 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.677659 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.677637 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sxls8\"" Jul 13 14:58:05.677828 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.677638 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 14:58:05.678742 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.678725 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 14:58:05.686042 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.686022 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng"] Jul 13 14:58:05.784448 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.784345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2630a39-f876-4377-b863-d43701d44e92-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.784448 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.784396 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmj8m\" (UniqueName: \"kubernetes.io/projected/b2630a39-f876-4377-b863-d43701d44e92-kube-api-access-vmj8m\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.784679 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.784528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2630a39-f876-4377-b863-d43701d44e92-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.785695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.785663 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:05.885532 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.885498 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2630a39-f876-4377-b863-d43701d44e92-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.885532 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.885536 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmj8m\" (UniqueName: \"kubernetes.io/projected/b2630a39-f876-4377-b863-d43701d44e92-kube-api-access-vmj8m\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.885817 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.885588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2630a39-f876-4377-b863-d43701d44e92-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.887905 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.887883 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2630a39-f876-4377-b863-d43701d44e92-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.888153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.888130 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2630a39-f876-4377-b863-d43701d44e92-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.902815 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.902770 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfjl4"] Jul 13 14:58:05.907893 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.907866 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmj8m\" (UniqueName: \"kubernetes.io/projected/b2630a39-f876-4377-b863-d43701d44e92-kube-api-access-vmj8m\") pod \"kube-auth-proxy-7cfcb69fbd-qktng\" (UID: \"b2630a39-f876-4377-b863-d43701d44e92\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.912037 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.911998 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:05.918060 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.918042 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 14:58:05.918389 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.918043 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6879n\"" Jul 13 14:58:05.919299 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.919243 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfjl4"] Jul 13 14:58:05.942529 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.942495 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:05.981710 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.981682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" Jul 13 14:58:05.986945 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.986922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b92b726-0be8-4ef0-a5a0-951fd9602788-cert\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:05.987081 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:05.986952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vmw\" (UniqueName: \"kubernetes.io/projected/1b92b726-0be8-4ef0-a5a0-951fd9602788-kube-api-access-t6vmw\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.028482 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.028389 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:06.087814 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.087447 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b92b726-0be8-4ef0-a5a0-951fd9602788-cert\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.087814 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.087510 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vmw\" (UniqueName: \"kubernetes.io/projected/1b92b726-0be8-4ef0-a5a0-951fd9602788-kube-api-access-t6vmw\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.089723 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.089699 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b92b726-0be8-4ef0-a5a0-951fd9602788-cert\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.098765 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.098738 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vmw\" (UniqueName: \"kubernetes.io/projected/1b92b726-0be8-4ef0-a5a0-951fd9602788-kube-api-access-t6vmw\") pod \"odh-model-controller-7998c67dc4-lfjl4\" (UID: \"1b92b726-0be8-4ef0-a5a0-951fd9602788\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.131059 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.131019 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:06.132825 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.132768 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng"] Jul 13 14:58:06.134548 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:58:06.134523 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2630a39_f876_4377_b863_d43701d44e92.slice/crio-85a9af52e30297964c1c9d60fe506b0cdbb648f055fc693b475cff8a48beedbd WatchSource:0}: Error finding container 85a9af52e30297964c1c9d60fe506b0cdbb648f055fc693b475cff8a48beedbd: Status 404 returned error can't find the container with id 85a9af52e30297964c1c9d60fe506b0cdbb648f055fc693b475cff8a48beedbd Jul 13 14:58:06.225639 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.225606 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:06.260550 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.260518 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" event={"ID":"b2630a39-f876-4377-b863-d43701d44e92","Type":"ContainerStarted","Data":"85a9af52e30297964c1c9d60fe506b0cdbb648f055fc693b475cff8a48beedbd"} Jul 13 14:58:06.390530 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.390506 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfjl4"] Jul 13 14:58:06.392636 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:58:06.392611 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b92b726_0be8_4ef0_a5a0_951fd9602788.slice/crio-085cd8fd415af986ed2fc39c417559ff7eba19090dd69a92a66000d01d126d38 WatchSource:0}: Error finding container 085cd8fd415af986ed2fc39c417559ff7eba19090dd69a92a66000d01d126d38: Status 404 returned error can't find the container with id 085cd8fd415af986ed2fc39c417559ff7eba19090dd69a92a66000d01d126d38 Jul 13 14:58:06.808167 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.808129 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fx4tk"] Jul 13 14:58:06.813135 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.813112 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:06.815763 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.815743 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 14:58:06.826011 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.825989 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fx4tk"] Jul 13 14:58:06.828877 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.828844 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:06.896661 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.896628 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:06.896856 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.896679 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jkg\" (UniqueName: \"kubernetes.io/projected/8798a704-483e-4d73-b938-df6e2bc2ea38-kube-api-access-k5jkg\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:06.997503 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.997377 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:06.997503 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.997430 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jkg\" (UniqueName: \"kubernetes.io/projected/8798a704-483e-4d73-b938-df6e2bc2ea38-kube-api-access-k5jkg\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:06.998069 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:58:06.997910 2554 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 14:58:06.998069 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:58:06.998028 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs podName:8798a704-483e-4d73-b938-df6e2bc2ea38 nodeName:}" failed. No retries permitted until 2026-07-13 14:58:07.497951676 +0000 UTC m=+350.388626771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs") pod "model-serving-api-54d89bd5fc-fx4tk" (UID: "8798a704-483e-4d73-b938-df6e2bc2ea38") : secret "model-serving-api-tls" not found Jul 13 14:58:06.999379 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.999300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:06.999982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:06.999948 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:07.003484 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.003464 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.006373 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.006354 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 14:58:07.006668 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.006651 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9gpjz\"" Jul 13 14:58:07.007017 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.006998 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 14:58:07.007119 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.007016 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 14:58:07.007405 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.007388 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 14:58:07.015594 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.015575 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jkg\" (UniqueName: \"kubernetes.io/projected/8798a704-483e-4d73-b938-df6e2bc2ea38-kube-api-access-k5jkg\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:07.021898 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.021448 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:07.093665 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.093586 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:07.098666 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098640 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.098760 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098710 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkk5h\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.098760 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098753 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.098901 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098802 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.098958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.098958 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098942 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.099061 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.098993 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.188923 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.188885 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:07.200529 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200472 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200545 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk5h\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200586 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200624 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200893 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200695 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200893 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200733 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.200893 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.200772 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.201466 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.201406 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.204142 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.204098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.204260 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.204172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.204394 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.204362 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.204394 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.204382 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.215545 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.215520 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.219200 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.219174 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk5h\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h\") pod \"istiod-openshift-gateway-7574459656-8xc6l\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.269256 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.269218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" event={"ID":"1b92b726-0be8-4ef0-a5a0-951fd9602788","Type":"ContainerStarted","Data":"085cd8fd415af986ed2fc39c417559ff7eba19090dd69a92a66000d01d126d38"} Jul 13 14:58:07.315995 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.315964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:07.504238 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.503717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:07.505982 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.505325 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:07.507932 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.507880 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8798a704-483e-4d73-b938-df6e2bc2ea38-tls-certs\") pod \"model-serving-api-54d89bd5fc-fx4tk\" (UID: \"8798a704-483e-4d73-b938-df6e2bc2ea38\") " pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:07.508874 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:58:07.508848 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ede54c_f9d1_41b4_8a53_48e1797d6368.slice/crio-75d79641a69bb5c13ebd6ed00ea920a1e00f6b05cb0244950582653b54b4fb5b WatchSource:0}: Error finding container 75d79641a69bb5c13ebd6ed00ea920a1e00f6b05cb0244950582653b54b4fb5b: Status 404 returned error can't find the container with id 75d79641a69bb5c13ebd6ed00ea920a1e00f6b05cb0244950582653b54b4fb5b Jul 13 14:58:07.726215 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.726132 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:07.925551 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:07.925452 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fx4tk"] Jul 13 14:58:07.935235 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:58:07.934942 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8798a704_483e_4d73_b938_df6e2bc2ea38.slice/crio-7f8a7995852141fb7e4df92a8c913f9ff411e2fb53314c1660ee0dd8f2ba9319 WatchSource:0}: Error finding container 7f8a7995852141fb7e4df92a8c913f9ff411e2fb53314c1660ee0dd8f2ba9319: Status 404 returned error can't find the container with id 7f8a7995852141fb7e4df92a8c913f9ff411e2fb53314c1660ee0dd8f2ba9319 Jul 13 14:58:08.244759 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:08.244723 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:08.277369 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:08.277288 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" event={"ID":"63ede54c-f9d1-41b4-8a53-48e1797d6368","Type":"ContainerStarted","Data":"75d79641a69bb5c13ebd6ed00ea920a1e00f6b05cb0244950582653b54b4fb5b"} Jul 13 14:58:08.279911 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:08.279835 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" event={"ID":"8798a704-483e-4d73-b938-df6e2bc2ea38","Type":"ContainerStarted","Data":"7f8a7995852141fb7e4df92a8c913f9ff411e2fb53314c1660ee0dd8f2ba9319"} Jul 13 14:58:11.831485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:11.831378 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 14:58:11.831485 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:11.831456 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 14:58:12.301710 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.301672 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" event={"ID":"8798a704-483e-4d73-b938-df6e2bc2ea38","Type":"ContainerStarted","Data":"32c41a387d6d68ecfe738fd1764477a721a382f2cd0853b2aa3f1dca66a55513"} Jul 13 14:58:12.301915 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.301738 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:12.303275 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.303248 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" event={"ID":"1b92b726-0be8-4ef0-a5a0-951fd9602788","Type":"ContainerStarted","Data":"34fe29e1d0d027d5662c3b891b57f9b64141e7b0decf6dd28bb4f9d63d70c8a9"} Jul 13 14:58:12.303406 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.303308 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:12.304780 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.304756 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" event={"ID":"b2630a39-f876-4377-b863-d43701d44e92","Type":"ContainerStarted","Data":"0a6720db6a9a07aaf37fd978a8f0e82f82254b069659aa0830237715996b2dff"} Jul 13 14:58:12.306428 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.306404 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" event={"ID":"63ede54c-f9d1-41b4-8a53-48e1797d6368","Type":"ContainerStarted","Data":"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25"} Jul 13 14:58:12.306612 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.306591 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:12.308448 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.308427 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-8xc6l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 14:58:12.308536 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.308473 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 14:58:12.320420 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.320364 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" podStartSLOduration=2.426060818 podStartE2EDuration="6.320348424s" totalImagesPullingTime="3.894287606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:12.319469828 +0000 UTC m=+355.210144931" watchObservedRunningTime="2026-07-13 14:58:12.320348424 +0000 UTC m=+355.211023527" Jul 13 14:58:12.341208 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.341157 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" podStartSLOduration=2.022421885 podStartE2EDuration="6.34113757s" totalImagesPullingTime="4.318715685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:12.338624755 +0000 UTC m=+355.229299858" watchObservedRunningTime="2026-07-13 14:58:12.34113757 +0000 UTC m=+355.231812673" Jul 13 14:58:12.358030 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.357970 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-qktng" podStartSLOduration=1.670213855 podStartE2EDuration="7.357949109s" totalImagesPullingTime="5.687735254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:12.356731507 +0000 UTC m=+355.247406635" watchObservedRunningTime="2026-07-13 14:58:12.357949109 +0000 UTC m=+355.248624212" Jul 13 14:58:12.378364 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.378300 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" podStartSLOduration=1.951158161 podStartE2EDuration="7.378278079s" totalImagesPullingTime="5.427119918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:12.376034508 +0000 UTC m=+355.266709612" watchObservedRunningTime="2026-07-13 14:58:12.378278079 +0000 UTC m=+355.268953182" Jul 13 14:58:12.421837 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.421445 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:12.513465 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:12.513435 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:13.183909 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:13.183871 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:13.311758 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:13.311729 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:13.344662 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:13.344628 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:13.443299 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:13.443225 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:17.777806 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.777760 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:17.880571 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.880532 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4"] Jul 13 14:58:17.885688 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.885662 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:17.888992 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.888962 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:17.889482 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.889041 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r8jxh\"" Jul 13 14:58:17.889482 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.889141 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 14:58:17.908092 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.908062 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4"] Jul 13 14:58:17.996645 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.996618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:17.996842 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:17.996657 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2htj\" (UniqueName: \"kubernetes.io/projected/5e828e4f-345b-4c09-aa57-f5fdb792d869-kube-api-access-s2htj\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.098294 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.098176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.098294 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.098234 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2htj\" (UniqueName: \"kubernetes.io/projected/5e828e4f-345b-4c09-aa57-f5fdb792d869-kube-api-access-s2htj\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.098528 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:58:18.098344 2554 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 14:58:18.098528 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:58:18.098419 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert podName:5e828e4f-345b-4c09-aa57-f5fdb792d869 nodeName:}" failed. No retries permitted until 2026-07-13 14:58:18.598396598 +0000 UTC m=+361.489071682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert") pod "llmisvc-controller-manager-777c6bd7d5-bj4g4" (UID: "5e828e4f-345b-4c09-aa57-f5fdb792d869") : secret "llmisvc-webhook-server-cert" not found Jul 13 14:58:18.103310 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.103244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:18.115826 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.115746 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2htj\" (UniqueName: \"kubernetes.io/projected/5e828e4f-345b-4c09-aa57-f5fdb792d869-kube-api-access-s2htj\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.273249 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.273213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:18.391483 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.391408 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:18.491404 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.491372 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:18.601149 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.601116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.603339 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.603317 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e828e4f-345b-4c09-aa57-f5fdb792d869-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj4g4\" (UID: \"5e828e4f-345b-4c09-aa57-f5fdb792d869\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.796971 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.796942 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:18.943467 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:18.943358 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4"] Jul 13 14:58:18.945536 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:58:18.945501 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5e828e4f_345b_4c09_aa57_f5fdb792d869.slice/crio-ed459294e5a2b84446094f6b4a0436ab5767f26f4997c79f8f035322cba93495 WatchSource:0}: Error finding container ed459294e5a2b84446094f6b4a0436ab5767f26f4997c79f8f035322cba93495: Status 404 returned error can't find the container with id ed459294e5a2b84446094f6b4a0436ab5767f26f4997c79f8f035322cba93495 Jul 13 14:58:19.337871 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:19.337836 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" event={"ID":"5e828e4f-345b-4c09-aa57-f5fdb792d869","Type":"ContainerStarted","Data":"ed459294e5a2b84446094f6b4a0436ab5767f26f4997c79f8f035322cba93495"} Jul 13 14:58:20.802480 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:20.802446 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:20.802960 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:20.802661 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerName="discovery" containerID="cri-o://f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25" gracePeriod=30 Jul 13 14:58:21.049662 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.049639 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:21.221712 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221611 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkk5h\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.221712 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221652 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.221712 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221681 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.222043 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221727 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.222043 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221872 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.222043 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221905 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.222043 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.221997 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap\") pod \"63ede54c-f9d1-41b4-8a53-48e1797d6368\" (UID: \"63ede54c-f9d1-41b4-8a53-48e1797d6368\") " Jul 13 14:58:21.222611 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.222541 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:58:21.224289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224239 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token" (OuterVolumeSpecName: "istio-token") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:58:21.224289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224263 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h" (OuterVolumeSpecName: "kube-api-access-kkk5h") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "kube-api-access-kkk5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:58:21.224425 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224336 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs" (OuterVolumeSpecName: "local-certs") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:58:21.224425 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224344 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:58:21.224425 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224382 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:58:21.224534 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.224503 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts" (OuterVolumeSpecName: "cacerts") pod "63ede54c-f9d1-41b4-8a53-48e1797d6368" (UID: "63ede54c-f9d1-41b4-8a53-48e1797d6368"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:58:21.323342 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323301 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkk5h\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-kube-api-access-kkk5h\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323342 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323340 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-dns-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323355 2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ede54c-f9d1-41b4-8a53-48e1797d6368-local-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323367 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323380 2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-kubeconfig\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323392 2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63ede54c-f9d1-41b4-8a53-48e1797d6368-cacerts\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.323527 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.323404 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63ede54c-f9d1-41b4-8a53-48e1797d6368-istio-csr-ca-configmap\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:58:21.349078 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.349041 2554 generic.go:356] "Generic (PLEG): container finished" podID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerID="f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25" exitCode=0 Jul 13 14:58:21.349235 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.349091 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" Jul 13 14:58:21.349235 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.349111 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" event={"ID":"63ede54c-f9d1-41b4-8a53-48e1797d6368","Type":"ContainerDied","Data":"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25"} Jul 13 14:58:21.349235 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.349161 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l" event={"ID":"63ede54c-f9d1-41b4-8a53-48e1797d6368","Type":"ContainerDied","Data":"75d79641a69bb5c13ebd6ed00ea920a1e00f6b05cb0244950582653b54b4fb5b"} Jul 13 14:58:21.349235 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.349183 2554 scope.go:117] "RemoveContainer" containerID="f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25" Jul 13 14:58:21.358972 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.358956 2554 scope.go:117] "RemoveContainer" containerID="f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25" Jul 13 14:58:21.359237 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:58:21.359221 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25\": container with ID starting with f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25 not found: ID does not exist" containerID="f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25" Jul 13 14:58:21.359289 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.359245 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25"} err="failed to get container status \"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25\": rpc error: code = NotFound desc = could not find container \"f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25\": container with ID starting with f957851ae3935952ae1218a5aa5aa4c9ed2f40edfdb7cf69fdaf88e6e59a6f25 not found: ID does not exist" Jul 13 14:58:21.379508 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.379482 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:21.384098 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.384074 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8xc6l"] Jul 13 14:58:21.680806 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:21.680739 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" path="/var/lib/kubelet/pods/63ede54c-f9d1-41b4-8a53-48e1797d6368/volumes" Jul 13 14:58:22.355026 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:22.354992 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" event={"ID":"5e828e4f-345b-4c09-aa57-f5fdb792d869","Type":"ContainerStarted","Data":"ad22c33d6360f62854e69626962fde6f8d42c3d897fcc29c4d7e0de1e1ad3af6"} Jul 13 14:58:22.355441 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:22.355099 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:58:22.372835 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:22.372757 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" podStartSLOduration=2.190121902 podStartE2EDuration="5.372740397s" totalImagesPullingTime="3.182618495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:22.372366629 +0000 UTC m=+365.263041742" watchObservedRunningTime="2026-07-13 14:58:22.372740397 +0000 UTC m=+365.263415501" Jul 13 14:58:23.313533 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:23.313499 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lfjl4" Jul 13 14:58:23.315584 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:23.315548 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-fx4tk" Jul 13 14:58:53.364446 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:58:53.364412 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj4g4" Jul 13 14:59:06.549387 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.549348 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:06.649758 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.649708 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:06.692264 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.692236 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:06.692575 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.692564 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerName="discovery" Jul 13 14:59:06.692622 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.692577 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerName="discovery" Jul 13 14:59:06.692660 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.692629 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ede54c-f9d1-41b4-8a53-48e1797d6368" containerName="discovery" Jul 13 14:59:06.705746 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.705712 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:06.705922 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.705827 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:06.708470 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.708443 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 14:59:06.709389 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.709371 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cl4m2\"" Jul 13 14:59:06.709507 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.709377 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 14:59:06.736526 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.736493 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:06.800481 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.800423 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrb52\" (UniqueName: \"kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52\") pod \"kuadrant-operator-catalog-mk9pq\" (UID: \"b9229733-3f8a-4744-96e4-05ec63297c30\") " pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:06.901199 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.901172 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrb52\" (UniqueName: \"kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52\") pod \"kuadrant-operator-catalog-mk9pq\" (UID: \"b9229733-3f8a-4744-96e4-05ec63297c30\") " pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:06.910010 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:06.909982 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrb52\" (UniqueName: \"kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52\") pod \"kuadrant-operator-catalog-mk9pq\" (UID: \"b9229733-3f8a-4744-96e4-05ec63297c30\") " pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:07.015913 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.015883 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:07.066399 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.065971 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:07.147552 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.147529 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:07.149767 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:07.149734 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9229733_3f8a_4744_96e4_05ec63297c30.slice/crio-a78df66e6c186b4a677b38ba352b23966bc96760e4dea41cf7b2e4a4f75cfe25 WatchSource:0}: Error finding container a78df66e6c186b4a677b38ba352b23966bc96760e4dea41cf7b2e4a4f75cfe25: Status 404 returned error can't find the container with id a78df66e6c186b4a677b38ba352b23966bc96760e4dea41cf7b2e4a4f75cfe25 Jul 13 14:59:07.271769 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.271729 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvzg2"] Jul 13 14:59:07.276401 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.276379 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:07.283099 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.283075 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvzg2"] Jul 13 14:59:07.405862 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.405770 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrj6m\" (UniqueName: \"kubernetes.io/projected/92c831c8-76fe-4e79-9504-e00004db8d93-kube-api-access-qrj6m\") pod \"kuadrant-operator-catalog-lvzg2\" (UID: \"92c831c8-76fe-4e79-9504-e00004db8d93\") " pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:07.504041 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.504008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:07.507070 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.507045 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrj6m\" (UniqueName: \"kubernetes.io/projected/92c831c8-76fe-4e79-9504-e00004db8d93-kube-api-access-qrj6m\") pod \"kuadrant-operator-catalog-lvzg2\" (UID: \"92c831c8-76fe-4e79-9504-e00004db8d93\") " pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:07.515303 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.515282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrj6m\" (UniqueName: \"kubernetes.io/projected/92c831c8-76fe-4e79-9504-e00004db8d93-kube-api-access-qrj6m\") pod \"kuadrant-operator-catalog-lvzg2\" (UID: \"92c831c8-76fe-4e79-9504-e00004db8d93\") " pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:07.556264 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.556232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" event={"ID":"b9229733-3f8a-4744-96e4-05ec63297c30","Type":"ContainerStarted","Data":"a78df66e6c186b4a677b38ba352b23966bc96760e4dea41cf7b2e4a4f75cfe25"} Jul 13 14:59:07.587553 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.587521 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:07.701834 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.701800 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:07.732443 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:07.732418 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lvzg2"] Jul 13 14:59:07.734197 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:07.734177 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c831c8_76fe_4e79_9504_e00004db8d93.slice/crio-bb80d0eca952bcde33614530584f148ebffd646a4e41f7eac1882b5436c616e2 WatchSource:0}: Error finding container bb80d0eca952bcde33614530584f148ebffd646a4e41f7eac1882b5436c616e2: Status 404 returned error can't find the container with id bb80d0eca952bcde33614530584f148ebffd646a4e41f7eac1882b5436c616e2 Jul 13 14:59:08.563659 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:08.563626 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" event={"ID":"92c831c8-76fe-4e79-9504-e00004db8d93","Type":"ContainerStarted","Data":"bb80d0eca952bcde33614530584f148ebffd646a4e41f7eac1882b5436c616e2"} Jul 13 14:59:09.570016 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.569983 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" event={"ID":"92c831c8-76fe-4e79-9504-e00004db8d93","Type":"ContainerStarted","Data":"fef37d17619e52ff59bf45f8a5757b99389fd185b549ad58d5c7c5130ea299b0"} Jul 13 14:59:09.571406 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.571378 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" event={"ID":"b9229733-3f8a-4744-96e4-05ec63297c30","Type":"ContainerStarted","Data":"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05"} Jul 13 14:59:09.571537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.571431 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" podUID="b9229733-3f8a-4744-96e4-05ec63297c30" containerName="registry-server" containerID="cri-o://b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05" gracePeriod=2 Jul 13 14:59:09.588617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.588574 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" podStartSLOduration=1.116415741 podStartE2EDuration="2.588560711s" totalImagesPullingTime="1.47214497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:09.587301657 +0000 UTC m=+412.477976770" watchObservedRunningTime="2026-07-13 14:59:09.588560711 +0000 UTC m=+412.479235812" Jul 13 14:59:09.605084 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.605045 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" podStartSLOduration=1.549246498 podStartE2EDuration="3.60503139s" totalImagesPullingTime="2.055784892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:09.603434983 +0000 UTC m=+412.494110098" watchObservedRunningTime="2026-07-13 14:59:09.60503139 +0000 UTC m=+412.495706470" Jul 13 14:59:09.823126 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.823067 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:09.926573 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.926536 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrb52\" (UniqueName: \"kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52\") pod \"b9229733-3f8a-4744-96e4-05ec63297c30\" (UID: \"b9229733-3f8a-4744-96e4-05ec63297c30\") " Jul 13 14:59:09.928729 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:09.928702 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52" (OuterVolumeSpecName: "kube-api-access-wrb52") pod "b9229733-3f8a-4744-96e4-05ec63297c30" (UID: "b9229733-3f8a-4744-96e4-05ec63297c30"). InnerVolumeSpecName "kube-api-access-wrb52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:59:10.027689 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.027655 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrb52\" (UniqueName: \"kubernetes.io/projected/b9229733-3f8a-4744-96e4-05ec63297c30-kube-api-access-wrb52\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 14:59:10.577475 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.577443 2554 generic.go:356] "Generic (PLEG): container finished" podID="b9229733-3f8a-4744-96e4-05ec63297c30" containerID="b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05" exitCode=0 Jul 13 14:59:10.577935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.577480 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" Jul 13 14:59:10.577935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.577519 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" event={"ID":"b9229733-3f8a-4744-96e4-05ec63297c30","Type":"ContainerDied","Data":"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05"} Jul 13 14:59:10.577935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.577564 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mk9pq" event={"ID":"b9229733-3f8a-4744-96e4-05ec63297c30","Type":"ContainerDied","Data":"a78df66e6c186b4a677b38ba352b23966bc96760e4dea41cf7b2e4a4f75cfe25"} Jul 13 14:59:10.577935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.577584 2554 scope.go:117] "RemoveContainer" containerID="b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05" Jul 13 14:59:10.588458 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.588444 2554 scope.go:117] "RemoveContainer" containerID="b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05" Jul 13 14:59:10.588726 ip-10-0-134-241 kubenswrapper[2554]: E0713 14:59:10.588700 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05\": container with ID starting with b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05 not found: ID does not exist" containerID="b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05" Jul 13 14:59:10.588811 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.588737 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05"} err="failed to get container status \"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05\": rpc error: code = NotFound desc = could not find container \"b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05\": container with ID starting with b02207c2f82b7b35fadad1e3a33c72c016063ba43cb5e2843f8418c41363be05 not found: ID does not exist" Jul 13 14:59:10.598292 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.598273 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:10.603660 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:10.603639 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mk9pq"] Jul 13 14:59:11.679525 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:11.679481 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9229733-3f8a-4744-96e4-05ec63297c30" path="/var/lib/kubelet/pods/b9229733-3f8a-4744-96e4-05ec63297c30/volumes" Jul 13 14:59:17.588470 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:17.588446 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:17.588840 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:17.588499 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:17.611983 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:17.611961 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:17.633999 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:17.633973 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lvzg2" Jul 13 14:59:26.241213 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.241179 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b84bd7d78-jgzmj"] Jul 13 14:59:26.241587 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.241505 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9229733-3f8a-4744-96e4-05ec63297c30" containerName="registry-server" Jul 13 14:59:26.241587 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.241515 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9229733-3f8a-4744-96e4-05ec63297c30" containerName="registry-server" Jul 13 14:59:26.241587 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.241579 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9229733-3f8a-4744-96e4-05ec63297c30" containerName="registry-server" Jul 13 14:59:26.245750 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.245727 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.249680 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.249652 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:59:26.249833 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.249652 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zvmr4\"" Jul 13 14:59:26.249833 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.249731 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:59:26.249965 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.249831 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:59:26.249965 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.249911 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:59:26.250083 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.250068 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:59:26.254432 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254405 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-trusted-ca-bundle\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254520 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254483 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-oauth-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254520 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxp7\" (UniqueName: \"kubernetes.io/projected/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-kube-api-access-dzxp7\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254595 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254543 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-oauth-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254657 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254641 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254695 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254673 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-service-ca\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254737 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254726 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.254771 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.254740 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 14:59:26.259862 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.259844 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b84bd7d78-jgzmj"] Jul 13 14:59:26.300614 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.300579 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:26.355687 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355647 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355687 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355693 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-service-ca\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355720 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355760 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-trusted-ca-bundle\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355837 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-oauth-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355865 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxp7\" (UniqueName: \"kubernetes.io/projected/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-kube-api-access-dzxp7\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.355974 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.355890 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-oauth-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.356545 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.356521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-service-ca\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.356545 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.356532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.356741 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.356721 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-trusted-ca-bundle\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.356913 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.356892 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-oauth-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.358515 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.358470 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-serving-cert\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.358618 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.358550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:26.358618 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.358580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-console-oauth-config\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.367060 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.367039 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxp7\" (UniqueName: \"kubernetes.io/projected/295dbd78-78e9-4ee6-8aeb-4d270d341ae9-kube-api-access-dzxp7\") pod \"console-b84bd7d78-jgzmj\" (UID: \"295dbd78-78e9-4ee6-8aeb-4d270d341ae9\") " pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.556723 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.556693 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:26.695499 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:26.695473 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b84bd7d78-jgzmj"] Jul 13 14:59:26.698021 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:26.697997 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295dbd78_78e9_4ee6_8aeb_4d270d341ae9.slice/crio-80a140c59a543730cff28a00beaea143cefe93b4368a76af548427a597947248 WatchSource:0}: Error finding container 80a140c59a543730cff28a00beaea143cefe93b4368a76af548427a597947248: Status 404 returned error can't find the container with id 80a140c59a543730cff28a00beaea143cefe93b4368a76af548427a597947248 Jul 13 14:59:27.657863 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:27.657833 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b84bd7d78-jgzmj" event={"ID":"295dbd78-78e9-4ee6-8aeb-4d270d341ae9","Type":"ContainerStarted","Data":"a4f05d6ecc98635cfa77d39fba702aeacab1a26558fca22348647d5603d2d951"} Jul 13 14:59:27.658311 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:27.657890 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b84bd7d78-jgzmj" event={"ID":"295dbd78-78e9-4ee6-8aeb-4d270d341ae9","Type":"ContainerStarted","Data":"80a140c59a543730cff28a00beaea143cefe93b4368a76af548427a597947248"} Jul 13 14:59:27.678176 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:27.678104 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b84bd7d78-jgzmj" podStartSLOduration=1.678086371 podStartE2EDuration="1.678086371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:27.675852375 +0000 UTC m=+430.566527477" watchObservedRunningTime="2026-07-13 14:59:27.678086371 +0000 UTC m=+430.568761473" Jul 13 14:59:32.479629 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:32.479590 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:35.100175 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.100143 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v"] Jul 13 14:59:35.103780 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.103762 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:35.106660 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.106639 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 14:59:35.106741 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.106681 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lvk4p\"" Jul 13 14:59:35.122272 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.122251 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v"] Jul 13 14:59:35.133964 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.133937 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdxs\" (UniqueName: \"kubernetes.io/projected/27bffcd5-5444-4e76-bc2a-d574dfa3768c-kube-api-access-jpdxs\") pod \"dns-operator-controller-manager-864dd78c4c-77s8v\" (UID: \"27bffcd5-5444-4e76-bc2a-d574dfa3768c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:35.234873 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.234840 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdxs\" (UniqueName: \"kubernetes.io/projected/27bffcd5-5444-4e76-bc2a-d574dfa3768c-kube-api-access-jpdxs\") pod \"dns-operator-controller-manager-864dd78c4c-77s8v\" (UID: \"27bffcd5-5444-4e76-bc2a-d574dfa3768c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:35.259468 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.259439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdxs\" (UniqueName: \"kubernetes.io/projected/27bffcd5-5444-4e76-bc2a-d574dfa3768c-kube-api-access-jpdxs\") pod \"dns-operator-controller-manager-864dd78c4c-77s8v\" (UID: \"27bffcd5-5444-4e76-bc2a-d574dfa3768c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:35.414251 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.414169 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:35.546345 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.546315 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v"] Jul 13 14:59:35.549652 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:35.549627 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bffcd5_5444_4e76_bc2a_d574dfa3768c.slice/crio-4b2a2d39d845965490703a4f15700ca6c2c1ba73e95bf667201a90b2dd9f7278 WatchSource:0}: Error finding container 4b2a2d39d845965490703a4f15700ca6c2c1ba73e95bf667201a90b2dd9f7278: Status 404 returned error can't find the container with id 4b2a2d39d845965490703a4f15700ca6c2c1ba73e95bf667201a90b2dd9f7278 Jul 13 14:59:35.695545 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:35.695464 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" event={"ID":"27bffcd5-5444-4e76-bc2a-d574dfa3768c","Type":"ContainerStarted","Data":"4b2a2d39d845965490703a4f15700ca6c2c1ba73e95bf667201a90b2dd9f7278"} Jul 13 14:59:36.558145 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.557491 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:36.558145 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.557563 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:36.562978 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.562952 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:36.707132 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.707104 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b84bd7d78-jgzmj" Jul 13 14:59:36.887286 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.887213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:36.993921 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:36.993892 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:37.100112 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:37.100074 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:37.708533 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:37.708501 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" event={"ID":"27bffcd5-5444-4e76-bc2a-d574dfa3768c","Type":"ContainerStarted","Data":"ae32b1eb876f4a6e987674c871ea2f67eb64ddff835a459b33e9297f3943dcff"} Jul 13 14:59:37.708997 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:37.708646 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:37.726996 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:37.726943 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" podStartSLOduration=0.698836457 podStartE2EDuration="2.726923666s" totalImagesPullingTime="2.028087209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:37.725510414 +0000 UTC m=+440.616185519" watchObservedRunningTime="2026-07-13 14:59:37.726923666 +0000 UTC m=+440.617598769" Jul 13 14:59:39.579465 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.579432 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx"] Jul 13 14:59:39.582869 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.582853 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:39.585919 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.585892 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-smh8p\"" Jul 13 14:59:39.604537 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.604481 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx"] Jul 13 14:59:39.676511 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.676485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdsqd\" (UniqueName: \"kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd\") pod \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" (UID: \"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:39.777765 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.777732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdsqd\" (UniqueName: \"kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd\") pod \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" (UID: \"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:39.789752 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.789726 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdsqd\" (UniqueName: \"kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd\") pod \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" (UID: \"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:39.893929 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:39.893834 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:40.031050 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:40.031024 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx"] Jul 13 14:59:40.033400 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:40.033372 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8a1225_60dd_4947_b7ef_8ac10190eb3a.slice/crio-f124d6f1d4dc0b4de74fce423e6e198d856c56de6c00743b49104a80ede433cc WatchSource:0}: Error finding container f124d6f1d4dc0b4de74fce423e6e198d856c56de6c00743b49104a80ede433cc: Status 404 returned error can't find the container with id f124d6f1d4dc0b4de74fce423e6e198d856c56de6c00743b49104a80ede433cc Jul 13 14:59:40.487889 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:40.487851 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:40.723152 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:40.723123 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" event={"ID":"ff8a1225-60dd-4947-b7ef-8ac10190eb3a","Type":"ContainerStarted","Data":"f124d6f1d4dc0b4de74fce423e6e198d856c56de6c00743b49104a80ede433cc"} Jul 13 14:59:42.732543 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:42.732501 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" event={"ID":"ff8a1225-60dd-4947-b7ef-8ac10190eb3a","Type":"ContainerStarted","Data":"ff0058ef6c605dd399676688f72e0d5f5acd835d1ec42a9faf272913b62c7f56"} Jul 13 14:59:42.732935 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:42.732570 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:42.752392 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:42.752346 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" podStartSLOduration=2.042066293 podStartE2EDuration="3.752332214s" totalImagesPullingTime="1.710265921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:42.750407744 +0000 UTC m=+445.641082836" watchObservedRunningTime="2026-07-13 14:59:42.752332214 +0000 UTC m=+445.643007316" Jul 13 14:59:45.667335 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.667302 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r"] Jul 13 14:59:45.672018 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.672001 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.674503 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.674484 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-76kpm\"" Jul 13 14:59:45.687018 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.686991 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r"] Jul 13 14:59:45.728262 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.728238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.728370 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.728300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4rh\" (UniqueName: \"kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.829677 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.829636 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.829890 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.829707 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4rh\" (UniqueName: \"kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.830079 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.830057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.845125 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.845101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4rh\" (UniqueName: \"kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:45.983617 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:45.983539 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:46.120771 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:46.120723 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r"] Jul 13 14:59:46.124988 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:46.124963 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c704eef_b23d_4a8d_b6c2_3c0c92cb6de7.slice/crio-553d5c6792d248bf7a06771f657e18933de64efc25dc60852d83e9026109f08a WatchSource:0}: Error finding container 553d5c6792d248bf7a06771f657e18933de64efc25dc60852d83e9026109f08a: Status 404 returned error can't find the container with id 553d5c6792d248bf7a06771f657e18933de64efc25dc60852d83e9026109f08a Jul 13 14:59:46.312644 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:46.312608 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:46.750142 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:46.750064 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" event={"ID":"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7","Type":"ContainerStarted","Data":"553d5c6792d248bf7a06771f657e18933de64efc25dc60852d83e9026109f08a"} Jul 13 14:59:47.111153 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:47.111122 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:48.195945 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:48.195907 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:48.715248 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:48.715215 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-77s8v" Jul 13 14:59:49.614118 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.614087 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr"] Jul 13 14:59:49.617032 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.617006 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:49.619681 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.619664 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dv5n5\"" Jul 13 14:59:49.627288 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.627263 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr"] Jul 13 14:59:49.763496 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.763458 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj46\" (UniqueName: \"kubernetes.io/projected/89acfeb7-902e-4d35-a33d-1d0742674e73-kube-api-access-rwj46\") pod \"authorino-operator-75cc8d5cb7-7mbwr\" (UID: \"89acfeb7-902e-4d35-a33d-1d0742674e73\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:49.864335 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.864254 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj46\" (UniqueName: \"kubernetes.io/projected/89acfeb7-902e-4d35-a33d-1d0742674e73-kube-api-access-rwj46\") pod \"authorino-operator-75cc8d5cb7-7mbwr\" (UID: \"89acfeb7-902e-4d35-a33d-1d0742674e73\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:49.880689 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.880655 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj46\" (UniqueName: \"kubernetes.io/projected/89acfeb7-902e-4d35-a33d-1d0742674e73-kube-api-access-rwj46\") pod \"authorino-operator-75cc8d5cb7-7mbwr\" (UID: \"89acfeb7-902e-4d35-a33d-1d0742674e73\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:49.928674 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:49.928643 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:50.083547 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:50.083518 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr"] Jul 13 14:59:50.086041 ip-10-0-134-241 kubenswrapper[2554]: W0713 14:59:50.086003 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89acfeb7_902e_4d35_a33d_1d0742674e73.slice/crio-968b84be3eaa5b5af6d522460996cb0b6ab211f4c5f51d1a7252174e32f2a7b4 WatchSource:0}: Error finding container 968b84be3eaa5b5af6d522460996cb0b6ab211f4c5f51d1a7252174e32f2a7b4: Status 404 returned error can't find the container with id 968b84be3eaa5b5af6d522460996cb0b6ab211f4c5f51d1a7252174e32f2a7b4 Jul 13 14:59:50.771984 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:50.771947 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" event={"ID":"89acfeb7-902e-4d35-a33d-1d0742674e73","Type":"ContainerStarted","Data":"968b84be3eaa5b5af6d522460996cb0b6ab211f4c5f51d1a7252174e32f2a7b4"} Jul 13 14:59:53.738514 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.738441 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 14:59:53.790487 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.790450 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" event={"ID":"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7","Type":"ContainerStarted","Data":"acb39d1fb2ef49e036cc06581c5ea7b5fd8b7f4629d2027e5b1c2e1f461623f8"} Jul 13 14:59:53.790669 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.790597 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 14:59:53.792124 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.792099 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" event={"ID":"89acfeb7-902e-4d35-a33d-1d0742674e73","Type":"ContainerStarted","Data":"90392a0b9d9a4f62af252f34c54ff229058fe3c9b4bf2f7da11a31f101b6ae56"} Jul 13 14:59:53.792238 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.792151 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 14:59:53.811842 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.811775 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" podStartSLOduration=1.670329643 podStartE2EDuration="8.811761852s" totalImagesPullingTime="7.141432209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:53.809405941 +0000 UTC m=+456.700081043" watchObservedRunningTime="2026-07-13 14:59:53.811761852 +0000 UTC m=+456.702437019" Jul 13 14:59:53.827651 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:53.827610 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" podStartSLOduration=1.606802948 podStartE2EDuration="4.827597033s" totalImagesPullingTime="3.220794085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:59:53.826241213 +0000 UTC m=+456.716916314" watchObservedRunningTime="2026-07-13 14:59:53.827597033 +0000 UTC m=+456.718272136" Jul 13 14:59:55.899106 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:55.899067 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:56.077051 ip-10-0-134-241 kubenswrapper[2554]: I0713 14:59:56.077008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:03.311838 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:03.311768 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:03.448745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:03.448710 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:04.799630 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:04.799598 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:04.800001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:04.799768 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7mbwr" Jul 13 15:00:05.118382 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:05.118296 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:05.959756 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:05.959723 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst"] Jul 13 15:00:05.964298 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:05.964282 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:05.978820 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:05.978780 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst"] Jul 13 15:00:06.043897 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.043862 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst"] Jul 13 15:00:06.044138 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.044117 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-22zm2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" Jul 13 15:00:06.082654 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.082624 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj"] Jul 13 15:00:06.085948 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.085931 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.100457 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.100431 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj"] Jul 13 15:00:06.112767 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.112734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.112953 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.112881 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zm2\" (UniqueName: \"kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.214111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.214011 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.214111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.214063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bf2\" (UniqueName: \"kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.214328 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.214117 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zm2\" (UniqueName: \"kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.214328 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.214145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.214448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.214424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.229537 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.229504 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zm2\" (UniqueName: \"kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.314984 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.314955 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bf2\" (UniqueName: \"kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.315163 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.315002 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.315351 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.315334 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.332511 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.332478 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bf2\" (UniqueName: \"kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2\") pod \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.396521 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.396487 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.403057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.403030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:06.544582 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.544550 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj"] Jul 13 15:00:06.548008 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:06.547978 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43904ed6_561e_4be4_9640_e56fc8ed510c.slice/crio-02a38b363aa6623085e548c6a7ab3c509a9d44f4b60c3db2685c284335b151ef WatchSource:0}: Error finding container 02a38b363aa6623085e548c6a7ab3c509a9d44f4b60c3db2685c284335b151ef: Status 404 returned error can't find the container with id 02a38b363aa6623085e548c6a7ab3c509a9d44f4b60c3db2685c284335b151ef Jul 13 15:00:06.586319 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.586292 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj"] Jul 13 15:00:06.593520 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.593496 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj"] Jul 13 15:00:06.606090 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.606065 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst"] Jul 13 15:00:06.616017 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.615987 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:06.616534 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.616512 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" containerName="manager" Jul 13 15:00:06.616534 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.616533 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" containerName="manager" Jul 13 15:00:06.616878 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.616640 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" containerName="manager" Jul 13 15:00:06.619714 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.619695 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst"] Jul 13 15:00:06.619817 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.619798 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.632388 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.632366 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:06.635279 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.635247 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r"] Jul 13 15:00:06.635529 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.635501 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" containerName="manager" containerID="cri-o://acb39d1fb2ef49e036cc06581c5ea7b5fd8b7f4629d2027e5b1c2e1f461623f8" gracePeriod=2 Jul 13 15:00:06.644812 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.644758 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r"] Jul 13 15:00:06.648047 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.648007 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.657334 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.657306 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx"] Jul 13 15:00:06.657519 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.657496 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" containerName="manager" containerID="cri-o://ff0058ef6c605dd399676688f72e0d5f5acd835d1ec42a9faf272913b62c7f56" gracePeriod=2 Jul 13 15:00:06.664932 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.664907 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:06.665452 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.665435 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" containerName="manager" Jul 13 15:00:06.665513 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.665455 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" containerName="manager" Jul 13 15:00:06.665549 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.665539 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" containerName="manager" Jul 13 15:00:06.668674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.668638 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.670474 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.670449 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx"] Jul 13 15:00:06.674262 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.674232 2554 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-p5mnx\": pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.676482 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.676455 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.678563 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.678539 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.680774 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.680754 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:06.699861 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.699824 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.718462 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.718434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.718622 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.718490 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjlf\" (UniqueName: \"kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.819722 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.819691 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjlf\" (UniqueName: \"kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.819911 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.819779 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.819986 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.819962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.820064 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.820048 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45g2\" (UniqueName: \"kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.820415 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.820398 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.834961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.834919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjlf\" (UniqueName: \"kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf\") pod \"kuadrant-operator-controller-manager-55467d9f7-dgvvm\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.867291 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.867136 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" containerName="manager" containerID="cri-o://aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf" gracePeriod=2 Jul 13 15:00:06.869047 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.869007 2554 generic.go:356] "Generic (PLEG): container finished" podID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" containerID="ff0058ef6c605dd399676688f72e0d5f5acd835d1ec42a9faf272913b62c7f56" exitCode=0 Jul 13 15:00:06.870179 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.870147 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.871348 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.871288 2554 generic.go:356] "Generic (PLEG): container finished" podID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" containerID="acb39d1fb2ef49e036cc06581c5ea7b5fd8b7f4629d2027e5b1c2e1f461623f8" exitCode=0 Jul 13 15:00:06.871471 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.871370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.872491 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.872463 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.885518 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.885485 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.890931 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.890903 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.901742 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.901714 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.908408 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.908372 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.915524 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.915501 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.919312 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.919294 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.921316 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.921298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45g2\" (UniqueName: \"kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.921400 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.921385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.921726 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.921711 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.921986 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.921968 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.925235 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.925217 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.928277 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.928253 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.931528 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.931510 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:06.932050 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.932030 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.944560 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.944538 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45g2\" (UniqueName: \"kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-psjxq\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:06.982951 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.982766 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.985581 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.985444 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.986451 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:06.986423 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.987858 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.987834 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:06.989998 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.989964 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:06.991959 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.991925 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.994392 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.994361 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:06.996313 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.996285 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:06.998536 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:06.998431 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.000619 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.000587 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:07.022190 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.021944 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22zm2\" (UniqueName: \"kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2\") pod \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " Jul 13 15:00:07.022190 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.022022 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume\") pod \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\" (UID: \"1bbde593-f8c7-4de1-8a64-1b65c719df3d\") " Jul 13 15:00:07.022429 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.022360 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1bbde593-f8c7-4de1-8a64-1b65c719df3d" (UID: "1bbde593-f8c7-4de1-8a64-1b65c719df3d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:07.024868 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.024835 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2" (OuterVolumeSpecName: "kube-api-access-22zm2") pod "1bbde593-f8c7-4de1-8a64-1b65c719df3d" (UID: "1bbde593-f8c7-4de1-8a64-1b65c719df3d"). InnerVolumeSpecName "kube-api-access-22zm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:07.084355 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.084317 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:07.087692 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.087667 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:07.094209 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:07.094177 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc082596b_747b_436e_b445_80a6dc43e3a7.slice/crio-aee93f845bd264cff865b3f61d6c3c0670722b80c0f7b431f86e7fccc2dd5d5a WatchSource:0}: Error finding container aee93f845bd264cff865b3f61d6c3c0670722b80c0f7b431f86e7fccc2dd5d5a: Status 404 returned error can't find the container with id aee93f845bd264cff865b3f61d6c3c0670722b80c0f7b431f86e7fccc2dd5d5a Jul 13 15:00:07.122674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.122650 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg4rh\" (UniqueName: \"kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh\") pod \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " Jul 13 15:00:07.122854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.122748 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume\") pod \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\" (UID: \"5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7\") " Jul 13 15:00:07.122854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.122773 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdsqd\" (UniqueName: \"kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd\") pod \"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\" (UID: \"ff8a1225-60dd-4947-b7ef-8ac10190eb3a\") " Jul 13 15:00:07.123040 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.123019 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zm2\" (UniqueName: \"kubernetes.io/projected/1bbde593-f8c7-4de1-8a64-1b65c719df3d-kube-api-access-22zm2\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.123096 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.123046 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbde593-f8c7-4de1-8a64-1b65c719df3d-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.123297 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.123275 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" (UID: "5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:07.123957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.123941 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.124746 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.124719 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh" (OuterVolumeSpecName: "kube-api-access-lg4rh") pod "5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" (UID: "5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7"). InnerVolumeSpecName "kube-api-access-lg4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:07.124831 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.124809 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd" (OuterVolumeSpecName: "kube-api-access-wdsqd") pod "ff8a1225-60dd-4947-b7ef-8ac10190eb3a" (UID: "ff8a1225-60dd-4947-b7ef-8ac10190eb3a"). InnerVolumeSpecName "kube-api-access-wdsqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:07.126552 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.126526 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:07.128536 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.128511 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:07.130416 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.130388 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:07.132452 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.132423 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.224456 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224413 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume\") pod \"43904ed6-561e-4be4-9640-e56fc8ed510c\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " Jul 13 15:00:07.224623 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224485 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5bf2\" (UniqueName: \"kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2\") pod \"43904ed6-561e-4be4-9640-e56fc8ed510c\" (UID: \"43904ed6-561e-4be4-9640-e56fc8ed510c\") " Jul 13 15:00:07.224714 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224691 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "43904ed6-561e-4be4-9640-e56fc8ed510c" (UID: "43904ed6-561e-4be4-9640-e56fc8ed510c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:07.224817 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224779 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4rh\" (UniqueName: \"kubernetes.io/projected/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-kube-api-access-lg4rh\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.224863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224820 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43904ed6-561e-4be4-9640-e56fc8ed510c-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.224863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224836 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.224863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.224848 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdsqd\" (UniqueName: \"kubernetes.io/projected/ff8a1225-60dd-4947-b7ef-8ac10190eb3a-kube-api-access-wdsqd\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.226473 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.226449 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2" (OuterVolumeSpecName: "kube-api-access-v5bf2") pod "43904ed6-561e-4be4-9640-e56fc8ed510c" (UID: "43904ed6-561e-4be4-9640-e56fc8ed510c"). InnerVolumeSpecName "kube-api-access-v5bf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:07.267474 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.267451 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:07.269924 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:07.269897 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710ea36f_a98c_41d1_9b2b_1bce191a33bd.slice/crio-166c7e54b61e701bb02e669ec7b1d0e8ac358bbee720f87bb2f421843c4e5854 WatchSource:0}: Error finding container 166c7e54b61e701bb02e669ec7b1d0e8ac358bbee720f87bb2f421843c4e5854: Status 404 returned error can't find the container with id 166c7e54b61e701bb02e669ec7b1d0e8ac358bbee720f87bb2f421843c4e5854 Jul 13 15:00:07.325720 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.325689 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5bf2\" (UniqueName: \"kubernetes.io/projected/43904ed6-561e-4be4-9640-e56fc8ed510c-kube-api-access-v5bf2\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.515466 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.515426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:07.677239 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.677148 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:07.678771 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.678742 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" path="/var/lib/kubelet/pods/1bbde593-f8c7-4de1-8a64-1b65c719df3d/volumes" Jul 13 15:00:07.679034 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.679020 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" path="/var/lib/kubelet/pods/43904ed6-561e-4be4-9640-e56fc8ed510c/volumes" Jul 13 15:00:07.679249 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.679229 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.679396 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.679382 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" path="/var/lib/kubelet/pods/5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7/volumes" Jul 13 15:00:07.679666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.679655 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" path="/var/lib/kubelet/pods/ff8a1225-60dd-4947-b7ef-8ac10190eb3a/volumes" Jul 13 15:00:07.681156 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.681129 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p5mnx\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:07.683029 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.683000 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h688r\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="5c704eef-b23d-4a8d-b6c2-3c0c92cb6de7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:07.877312 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.877287 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h688r" Jul 13 15:00:07.877471 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.877320 2554 scope.go:117] "RemoveContainer" containerID="acb39d1fb2ef49e036cc06581c5ea7b5fd8b7f4629d2027e5b1c2e1f461623f8" Jul 13 15:00:07.878828 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.878778 2554 generic.go:356] "Generic (PLEG): container finished" podID="43904ed6-561e-4be4-9640-e56fc8ed510c" containerID="aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf" exitCode=2 Jul 13 15:00:07.878921 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.878827 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.880983 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.880956 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" event={"ID":"c082596b-747b-436e-b445-80a6dc43e3a7","Type":"ContainerStarted","Data":"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518"} Jul 13 15:00:07.881103 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.881009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" event={"ID":"c082596b-747b-436e-b445-80a6dc43e3a7","Type":"ContainerStarted","Data":"aee93f845bd264cff865b3f61d6c3c0670722b80c0f7b431f86e7fccc2dd5d5a"} Jul 13 15:00:07.881207 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.881191 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:07.882515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.882408 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p5mnx" Jul 13 15:00:07.884331 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.884311 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" event={"ID":"710ea36f-a98c-41d1-9b2b-1bce191a33bd","Type":"ContainerStarted","Data":"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85"} Jul 13 15:00:07.884408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.884353 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" event={"ID":"710ea36f-a98c-41d1-9b2b-1bce191a33bd","Type":"ContainerStarted","Data":"166c7e54b61e701bb02e669ec7b1d0e8ac358bbee720f87bb2f421843c4e5854"} Jul 13 15:00:07.884408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.884382 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:07.884408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.884321 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:07.887636 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.887619 2554 scope.go:117] "RemoveContainer" containerID="aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf" Jul 13 15:00:07.895159 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.895139 2554 scope.go:117] "RemoveContainer" containerID="aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf" Jul 13 15:00:07.895409 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.895384 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf\": container with ID starting with aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf not found: ID does not exist" containerID="aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf" Jul 13 15:00:07.895494 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.895419 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf"} err="failed to get container status \"aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf\": rpc error: code = NotFound desc = could not find container \"aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf\": container with ID starting with aadfa4314290ff704053c10bf4abda7b6ab61717de46cdd8a5a83f2365b75caf not found: ID does not exist" Jul 13 15:00:07.895494 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.895441 2554 scope.go:117] "RemoveContainer" containerID="ff0058ef6c605dd399676688f72e0d5f5acd835d1ec42a9faf272913b62c7f56" Jul 13 15:00:07.902282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.902218 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" podStartSLOduration=1.90220569 podStartE2EDuration="1.90220569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:07.901279718 +0000 UTC m=+470.791954821" watchObservedRunningTime="2026-07-13 15:00:07.90220569 +0000 UTC m=+470.792880878" Jul 13 15:00:07.903446 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.903371 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-k6srj\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="43904ed6-561e-4be4-9640-e56fc8ed510c" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-k6srj" Jul 13 15:00:07.905395 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:07.905370 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qhrst\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="1bbde593-f8c7-4de1-8a64-1b65c719df3d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qhrst" Jul 13 15:00:07.930629 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:07.930528 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" podStartSLOduration=1.9305111780000002 podStartE2EDuration="1.930511178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:07.927920653 +0000 UTC m=+470.818595749" watchObservedRunningTime="2026-07-13 15:00:07.930511178 +0000 UTC m=+470.821186279" Jul 13 15:00:08.662182 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:08.662145 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:08.762314 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:08.762279 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:09.130131 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:09.130100 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:09.230261 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:09.230227 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:13.704185 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:13.704141 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:13.804557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:13.804521 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:16.864426 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:16.864385 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:16.965604 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:16.965566 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:18.894523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:18.894496 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:18.894915 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:18.894556 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:18.965310 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:18.965274 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:18.965506 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:18.965476 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" podUID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" containerName="manager" containerID="cri-o://d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85" gracePeriod=10 Jul 13 15:00:19.215390 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.215358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:19.220035 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.220013 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:00:19.220394 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.220373 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" containerName="manager" Jul 13 15:00:19.220443 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.220396 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" containerName="manager" Jul 13 15:00:19.220476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.220466 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8a1225-60dd-4947-b7ef-8ac10190eb3a" containerName="manager" Jul 13 15:00:19.220547 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.220465 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:19.223893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.223879 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.238592 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.238568 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:00:19.324751 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.324704 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume\") pod \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " Jul 13 15:00:19.324952 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.324829 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l45g2\" (UniqueName: \"kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2\") pod \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\" (UID: \"710ea36f-a98c-41d1-9b2b-1bce191a33bd\") " Jul 13 15:00:19.325050 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.325020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.325111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.325089 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnzw\" (UniqueName: \"kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.325176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.325147 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "710ea36f-a98c-41d1-9b2b-1bce191a33bd" (UID: "710ea36f-a98c-41d1-9b2b-1bce191a33bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:19.325249 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.325227 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/710ea36f-a98c-41d1-9b2b-1bce191a33bd-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:19.326778 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.326753 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2" (OuterVolumeSpecName: "kube-api-access-l45g2") pod "710ea36f-a98c-41d1-9b2b-1bce191a33bd" (UID: "710ea36f-a98c-41d1-9b2b-1bce191a33bd"). InnerVolumeSpecName "kube-api-access-l45g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:19.426439 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.426402 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.426599 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.426451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnzw\" (UniqueName: \"kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.426599 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.426517 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l45g2\" (UniqueName: \"kubernetes.io/projected/710ea36f-a98c-41d1-9b2b-1bce191a33bd-kube-api-access-l45g2\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:19.426902 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.426880 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.440302 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.440272 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnzw\" (UniqueName: \"kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw\") pod \"kuadrant-operator-controller-manager-86878c769f-d5526\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.532778 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.532749 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.673017 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.672995 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:00:19.675271 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:19.675246 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda352df7f_ad62_427d_9531_c7c1c0febf22.slice/crio-566232b588884a5b8fb848d5f3f0735cdc305b4c4efd5f38396bcbd9a2134dfd WatchSource:0}: Error finding container 566232b588884a5b8fb848d5f3f0735cdc305b4c4efd5f38396bcbd9a2134dfd: Status 404 returned error can't find the container with id 566232b588884a5b8fb848d5f3f0735cdc305b4c4efd5f38396bcbd9a2134dfd Jul 13 15:00:19.947378 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.947294 2554 generic.go:356] "Generic (PLEG): container finished" podID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" containerID="d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85" exitCode=0 Jul 13 15:00:19.947378 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.947355 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" Jul 13 15:00:19.947893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.947374 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" event={"ID":"710ea36f-a98c-41d1-9b2b-1bce191a33bd","Type":"ContainerDied","Data":"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85"} Jul 13 15:00:19.947893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.947424 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq" event={"ID":"710ea36f-a98c-41d1-9b2b-1bce191a33bd","Type":"ContainerDied","Data":"166c7e54b61e701bb02e669ec7b1d0e8ac358bbee720f87bb2f421843c4e5854"} Jul 13 15:00:19.947893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.947439 2554 scope.go:117] "RemoveContainer" containerID="d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85" Jul 13 15:00:19.949268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.949241 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" event={"ID":"a352df7f-ad62-427d-9531-c7c1c0febf22","Type":"ContainerStarted","Data":"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e"} Jul 13 15:00:19.949397 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.949286 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" event={"ID":"a352df7f-ad62-427d-9531-c7c1c0febf22","Type":"ContainerStarted","Data":"566232b588884a5b8fb848d5f3f0735cdc305b4c4efd5f38396bcbd9a2134dfd"} Jul 13 15:00:19.949397 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.949317 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:19.956093 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.956074 2554 scope.go:117] "RemoveContainer" containerID="d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85" Jul 13 15:00:19.956338 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:19.956321 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85\": container with ID starting with d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85 not found: ID does not exist" containerID="d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85" Jul 13 15:00:19.956381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.956345 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85"} err="failed to get container status \"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85\": rpc error: code = NotFound desc = could not find container \"d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85\": container with ID starting with d074ac12bbc7b43abecd4d800235fb4335de8a0c837ad03556d9f5c485f73c85 not found: ID does not exist" Jul 13 15:00:19.976344 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.976319 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:19.984720 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:19.984700 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-psjxq"] Jul 13 15:00:20.006554 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:20.006514 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" podStartSLOduration=1.006500351 podStartE2EDuration="1.006500351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:20.005876085 +0000 UTC m=+482.896551188" watchObservedRunningTime="2026-07-13 15:00:20.006500351 +0000 UTC m=+482.897175452" Jul 13 15:00:20.258563 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:20.258484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:20.359269 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:20.359230 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:21.679415 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:21.679379 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" path="/var/lib/kubelet/pods/710ea36f-a98c-41d1-9b2b-1bce191a33bd/volumes" Jul 13 15:00:28.917608 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:28.917550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:29.017712 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:29.017674 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:30.956945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:30.956908 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:00:31.009887 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.009857 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:31.010126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.010086 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" podUID="c082596b-747b-436e-b445-80a6dc43e3a7" containerName="manager" containerID="cri-o://803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518" gracePeriod=10 Jul 13 15:00:31.265267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.265243 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:31.322678 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.322647 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume\") pod \"c082596b-747b-436e-b445-80a6dc43e3a7\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " Jul 13 15:00:31.322678 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.322684 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxjlf\" (UniqueName: \"kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf\") pod \"c082596b-747b-436e-b445-80a6dc43e3a7\" (UID: \"c082596b-747b-436e-b445-80a6dc43e3a7\") " Jul 13 15:00:31.323033 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.323011 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c082596b-747b-436e-b445-80a6dc43e3a7" (UID: "c082596b-747b-436e-b445-80a6dc43e3a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:31.324710 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.324688 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf" (OuterVolumeSpecName: "kube-api-access-jxjlf") pod "c082596b-747b-436e-b445-80a6dc43e3a7" (UID: "c082596b-747b-436e-b445-80a6dc43e3a7"). InnerVolumeSpecName "kube-api-access-jxjlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:31.424356 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.424322 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082596b-747b-436e-b445-80a6dc43e3a7-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:31.424356 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.424352 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjlf\" (UniqueName: \"kubernetes.io/projected/c082596b-747b-436e-b445-80a6dc43e3a7-kube-api-access-jxjlf\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:31.534520 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.534490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:31.635744 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:31.635702 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:32.007149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.007072 2554 generic.go:356] "Generic (PLEG): container finished" podID="c082596b-747b-436e-b445-80a6dc43e3a7" containerID="803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518" exitCode=0 Jul 13 15:00:32.007149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.007114 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" Jul 13 15:00:32.007601 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.007118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" event={"ID":"c082596b-747b-436e-b445-80a6dc43e3a7","Type":"ContainerDied","Data":"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518"} Jul 13 15:00:32.007601 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.007218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm" event={"ID":"c082596b-747b-436e-b445-80a6dc43e3a7","Type":"ContainerDied","Data":"aee93f845bd264cff865b3f61d6c3c0670722b80c0f7b431f86e7fccc2dd5d5a"} Jul 13 15:00:32.007601 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.007239 2554 scope.go:117] "RemoveContainer" containerID="803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518" Jul 13 15:00:32.015705 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.015517 2554 scope.go:117] "RemoveContainer" containerID="803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518" Jul 13 15:00:32.015821 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:32.015778 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518\": container with ID starting with 803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518 not found: ID does not exist" containerID="803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518" Jul 13 15:00:32.015902 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.015829 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518"} err="failed to get container status \"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518\": rpc error: code = NotFound desc = could not find container \"803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518\": container with ID starting with 803de467edc1d2f3ca874bd00d0743e4910c7639a082b1af678952be19e70518 not found: ID does not exist" Jul 13 15:00:32.027301 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.027281 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:32.033833 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:32.033814 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dgvvm"] Jul 13 15:00:33.678146 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:33.678112 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c082596b-747b-436e-b445-80a6dc43e3a7" path="/var/lib/kubelet/pods/c082596b-747b-436e-b445-80a6dc43e3a7/volumes" Jul 13 15:00:34.722312 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:34.722269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:34.821859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:34.821825 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:35.399665 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.399631 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2"] Jul 13 15:00:35.400057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400042 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" containerName="manager" Jul 13 15:00:35.400126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400059 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" containerName="manager" Jul 13 15:00:35.400126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400078 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c082596b-747b-436e-b445-80a6dc43e3a7" containerName="manager" Jul 13 15:00:35.400126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400084 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c082596b-747b-436e-b445-80a6dc43e3a7" containerName="manager" Jul 13 15:00:35.400226 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400147 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="710ea36f-a98c-41d1-9b2b-1bce191a33bd" containerName="manager" Jul 13 15:00:35.400226 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.400156 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c082596b-747b-436e-b445-80a6dc43e3a7" containerName="manager" Jul 13 15:00:35.402773 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.402757 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.405574 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.405554 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 15:00:35.405696 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.405585 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nfm9p\"" Jul 13 15:00:35.416167 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.416142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2"] Jul 13 15:00:35.459652 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459619 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.459846 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459679 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bece848d-6294-46cc-b9da-d2e5f1436678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.459846 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459725 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.459846 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459751 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.460020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.460020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459903 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.460020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.460020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.459976 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwxx\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-kube-api-access-dbwxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.460020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.460013 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bece848d-6294-46cc-b9da-d2e5f1436678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561278 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561377 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561428 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwxx\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-kube-api-access-dbwxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561457 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bece848d-6294-46cc-b9da-d2e5f1436678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561755 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561865 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bece848d-6294-46cc-b9da-d2e5f1436678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.561961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561940 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.562080 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.561956 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.562080 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.562036 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.562381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.562360 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bece848d-6294-46cc-b9da-d2e5f1436678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.563890 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.563868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bece848d-6294-46cc-b9da-d2e5f1436678-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.564190 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.564169 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bece848d-6294-46cc-b9da-d2e5f1436678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.570183 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.570156 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.570265 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.570236 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwxx\" (UniqueName: \"kubernetes.io/projected/bece848d-6294-46cc-b9da-d2e5f1436678-kube-api-access-dbwxx\") pod \"maas-default-gateway-openshift-default-56654dcf7-wggx2\" (UID: \"bece848d-6294-46cc-b9da-d2e5f1436678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.654348 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.654251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:35.714437 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.714401 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:35.753210 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.753167 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:35.851975 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:35.851955 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2"] Jul 13 15:00:35.854227 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:35.854202 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbece848d_6294_46cc_b9da_d2e5f1436678.slice/crio-15a15a0b9b47ac1fd45b8d243d9a2bc42dbf413cfe7bba1dc762abea22a86408 WatchSource:0}: Error finding container 15a15a0b9b47ac1fd45b8d243d9a2bc42dbf413cfe7bba1dc762abea22a86408: Status 404 returned error can't find the container with id 15a15a0b9b47ac1fd45b8d243d9a2bc42dbf413cfe7bba1dc762abea22a86408 Jul 13 15:00:36.028399 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:36.028365 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" event={"ID":"bece848d-6294-46cc-b9da-d2e5f1436678","Type":"ContainerStarted","Data":"15a15a0b9b47ac1fd45b8d243d9a2bc42dbf413cfe7bba1dc762abea22a86408"} Jul 13 15:00:38.238995 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:38.238956 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:38.340540 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:38.340503 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:38.451761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:38.451728 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:00:38.451861 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:38.451823 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:00:38.451900 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:38.451865 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:00:39.047214 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:39.047183 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" event={"ID":"bece848d-6294-46cc-b9da-d2e5f1436678","Type":"ContainerStarted","Data":"1c7643ebd9cc97614347bd624f37ea2a763a98476262992c4c7081333072e202"} Jul 13 15:00:39.080677 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:39.080625 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podStartSLOduration=1.485209815 podStartE2EDuration="4.080608621s" totalImagesPullingTime="2.595398806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:39.07833107 +0000 UTC m=+501.969006171" watchObservedRunningTime="2026-07-13 15:00:39.080608621 +0000 UTC m=+501.971283723" Jul 13 15:00:39.714878 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:39.714843 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:39.719897 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:39.719867 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:40.052550 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:40.052516 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:40.053340 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:40.053317 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:00:40.188295 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:40.188249 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:48.120914 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:48.120875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:48.223843 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:48.223798 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:48.322934 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:48.322899 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:48.397825 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:48.397721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.083340 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.083295 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.095997 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.095968 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:00:50.098513 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.098492 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.101612 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.101591 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 15:00:50.102749 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.102722 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ndlxh\"" Jul 13 15:00:50.109660 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.109637 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:00:50.194970 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.194936 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvsl\" (UniqueName: \"kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.195172 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.195081 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.195814 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.195753 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.197316 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.197296 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:00:50.293880 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.293833 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.295919 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.295896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.296055 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.295940 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvsl\" (UniqueName: \"kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.296820 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.296772 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.304958 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.304926 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvsl\" (UniqueName: \"kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl\") pod \"limitador-limitador-64c94fd64f-nwql8\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.409311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.409222 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:50.570170 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:50.570148 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:00:50.572454 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:50.572429 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c89308_baab_4405_abc0_73c971bf7c09.slice/crio-1f6304ad3dcadc4a47f533865715fff2200b41fab644eec255df9f158eae75a9 WatchSource:0}: Error finding container 1f6304ad3dcadc4a47f533865715fff2200b41fab644eec255df9f158eae75a9: Status 404 returned error can't find the container with id 1f6304ad3dcadc4a47f533865715fff2200b41fab644eec255df9f158eae75a9 Jul 13 15:00:51.017343 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.017311 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:51.020917 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.020895 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:51.023691 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.023676 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xhtrg\"" Jul 13 15:00:51.028570 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.028487 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:51.053877 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.053844 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:51.108564 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.105585 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmzw\" (UniqueName: \"kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw\") pod \"authorino-58cd955675-whzss\" (UID: \"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c\") " pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:51.112709 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.112668 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" event={"ID":"f3c89308-baab-4405-abc0-73c971bf7c09","Type":"ContainerStarted","Data":"1f6304ad3dcadc4a47f533865715fff2200b41fab644eec255df9f158eae75a9"} Jul 13 15:00:51.113558 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.113490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:51.206316 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.206281 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmzw\" (UniqueName: \"kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw\") pod \"authorino-58cd955675-whzss\" (UID: \"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c\") " pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:51.224104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.224073 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmzw\" (UniqueName: \"kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw\") pod \"authorino-58cd955675-whzss\" (UID: \"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c\") " pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:51.331130 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.331059 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:51.474887 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:51.474858 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:51.477360 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:00:51.477323 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f5b5d1_2112_40f7_a3bb_8281a0813f9c.slice/crio-4e8050e3130662f7b83622a7b51bc8085cfa57f3c38f137094cd3c05b981ae13 WatchSource:0}: Error finding container 4e8050e3130662f7b83622a7b51bc8085cfa57f3c38f137094cd3c05b981ae13: Status 404 returned error can't find the container with id 4e8050e3130662f7b83622a7b51bc8085cfa57f3c38f137094cd3c05b981ae13 Jul 13 15:00:52.120153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:52.120116 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whzss" event={"ID":"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c","Type":"ContainerStarted","Data":"4e8050e3130662f7b83622a7b51bc8085cfa57f3c38f137094cd3c05b981ae13"} Jul 13 15:00:56.145638 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:56.145592 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whzss" event={"ID":"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c","Type":"ContainerStarted","Data":"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4"} Jul 13 15:00:56.147524 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:56.147499 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" event={"ID":"f3c89308-baab-4405-abc0-73c971bf7c09","Type":"ContainerStarted","Data":"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486"} Jul 13 15:00:56.147668 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:56.147589 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:00:56.164211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:56.164169 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-whzss" podStartSLOduration=2.3359527780000002 podStartE2EDuration="6.164156994s" totalImagesPullingTime="3.828204216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:56.163462333 +0000 UTC m=+519.054137436" watchObservedRunningTime="2026-07-13 15:00:56.164156994 +0000 UTC m=+519.054832096" Jul 13 15:00:58.443821 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.443743 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" podStartSLOduration=3.774513122 podStartE2EDuration="8.443719346s" totalImagesPullingTime="4.669206224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:56.184090954 +0000 UTC m=+519.074766059" watchObservedRunningTime="2026-07-13 15:00:58.443719346 +0000 UTC m=+521.334394450" Jul 13 15:00:58.446330 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.446301 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:58.446521 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.446496 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-whzss" podUID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" containerName="authorino" containerID="cri-o://8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4" gracePeriod=30 Jul 13 15:00:58.695652 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.695591 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:58.877896 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.877863 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gmzw\" (UniqueName: \"kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw\") pod \"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c\" (UID: \"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c\") " Jul 13 15:00:58.879917 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.879887 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw" (OuterVolumeSpecName: "kube-api-access-9gmzw") pod "c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" (UID: "c8f5b5d1-2112-40f7-a3bb-8281a0813f9c"). InnerVolumeSpecName "kube-api-access-9gmzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:58.978936 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:58.978864 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmzw\" (UniqueName: \"kubernetes.io/projected/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c-kube-api-access-9gmzw\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:00:59.169739 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.169705 2554 generic.go:356] "Generic (PLEG): container finished" podID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" containerID="8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4" exitCode=0 Jul 13 15:00:59.169945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.169746 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whzss" Jul 13 15:00:59.169945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.169814 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whzss" event={"ID":"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c","Type":"ContainerDied","Data":"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4"} Jul 13 15:00:59.169945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.169858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whzss" event={"ID":"c8f5b5d1-2112-40f7-a3bb-8281a0813f9c","Type":"ContainerDied","Data":"4e8050e3130662f7b83622a7b51bc8085cfa57f3c38f137094cd3c05b981ae13"} Jul 13 15:00:59.169945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.169882 2554 scope.go:117] "RemoveContainer" containerID="8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4" Jul 13 15:00:59.178662 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.178645 2554 scope.go:117] "RemoveContainer" containerID="8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4" Jul 13 15:00:59.178921 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:00:59.178905 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4\": container with ID starting with 8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4 not found: ID does not exist" containerID="8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4" Jul 13 15:00:59.178995 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.178929 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4"} err="failed to get container status \"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4\": rpc error: code = NotFound desc = could not find container \"8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4\": container with ID starting with 8bb8e237725b71b263440f37fca4166f91db707299bf079fd683623a6d9c7bd4 not found: ID does not exist" Jul 13 15:00:59.191477 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.191456 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:59.195514 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.195497 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-whzss"] Jul 13 15:00:59.679542 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:00:59.679510 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" path="/var/lib/kubelet/pods/c8f5b5d1-2112-40f7-a3bb-8281a0813f9c/volumes" Jul 13 15:01:06.545386 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:06.545301 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:01:06.546012 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:06.545567 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" podUID="f3c89308-baab-4405-abc0-73c971bf7c09" containerName="limitador" containerID="cri-o://d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486" gracePeriod=30 Jul 13 15:01:06.546427 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:06.546285 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:01:07.098213 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.098189 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:01:07.141745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.141677 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxvsl\" (UniqueName: \"kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl\") pod \"f3c89308-baab-4405-abc0-73c971bf7c09\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " Jul 13 15:01:07.141745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.141705 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file\") pod \"f3c89308-baab-4405-abc0-73c971bf7c09\" (UID: \"f3c89308-baab-4405-abc0-73c971bf7c09\") " Jul 13 15:01:07.142100 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.142076 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file" (OuterVolumeSpecName: "config-file") pod "f3c89308-baab-4405-abc0-73c971bf7c09" (UID: "f3c89308-baab-4405-abc0-73c971bf7c09"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:01:07.143661 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.143636 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl" (OuterVolumeSpecName: "kube-api-access-qxvsl") pod "f3c89308-baab-4405-abc0-73c971bf7c09" (UID: "f3c89308-baab-4405-abc0-73c971bf7c09"). InnerVolumeSpecName "kube-api-access-qxvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:01:07.211143 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.211108 2554 generic.go:356] "Generic (PLEG): container finished" podID="f3c89308-baab-4405-abc0-73c971bf7c09" containerID="d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486" exitCode=0 Jul 13 15:01:07.211308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.211156 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" Jul 13 15:01:07.211308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.211194 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" event={"ID":"f3c89308-baab-4405-abc0-73c971bf7c09","Type":"ContainerDied","Data":"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486"} Jul 13 15:01:07.211308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.211240 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nwql8" event={"ID":"f3c89308-baab-4405-abc0-73c971bf7c09","Type":"ContainerDied","Data":"1f6304ad3dcadc4a47f533865715fff2200b41fab644eec255df9f158eae75a9"} Jul 13 15:01:07.211308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.211261 2554 scope.go:117] "RemoveContainer" containerID="d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486" Jul 13 15:01:07.220236 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.220219 2554 scope.go:117] "RemoveContainer" containerID="d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486" Jul 13 15:01:07.220492 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:01:07.220477 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486\": container with ID starting with d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486 not found: ID does not exist" containerID="d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486" Jul 13 15:01:07.220537 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.220502 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486"} err="failed to get container status \"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486\": rpc error: code = NotFound desc = could not find container \"d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486\": container with ID starting with d2166fc86afc2d8f00da861b18d2e291eac931ac5a8e1682f80e659c23e2e486 not found: ID does not exist" Jul 13 15:01:07.235951 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.235922 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:01:07.240268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.240247 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nwql8"] Jul 13 15:01:07.242955 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.242938 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvsl\" (UniqueName: \"kubernetes.io/projected/f3c89308-baab-4405-abc0-73c971bf7c09-kube-api-access-qxvsl\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:01:07.243016 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.242959 2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3c89308-baab-4405-abc0-73c971bf7c09-config-file\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:01:07.679002 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:07.678961 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c89308-baab-4405-abc0-73c971bf7c09" path="/var/lib/kubelet/pods/f3c89308-baab-4405-abc0-73c971bf7c09/volumes" Jul 13 15:01:11.695939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:11.695863 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:11.794331 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:11.794291 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:12.240109 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240083 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m"] Jul 13 15:01:12.240444 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240432 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" containerName="authorino" Jul 13 15:01:12.240489 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240445 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" containerName="authorino" Jul 13 15:01:12.240489 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240468 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c89308-baab-4405-abc0-73c971bf7c09" containerName="limitador" Jul 13 15:01:12.240489 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240474 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c89308-baab-4405-abc0-73c971bf7c09" containerName="limitador" Jul 13 15:01:12.240581 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240530 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f5b5d1-2112-40f7-a3bb-8281a0813f9c" containerName="authorino" Jul 13 15:01:12.240581 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.240539 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c89308-baab-4405-abc0-73c971bf7c09" containerName="limitador" Jul 13 15:01:12.243584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.243569 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.246716 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.246697 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 13 15:01:12.247722 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.247706 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-k256t\"" Jul 13 15:01:12.247937 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.247924 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 15:01:12.248670 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.248652 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 15:01:12.257701 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.257680 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m"] Jul 13 15:01:12.291997 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.291970 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f176e3d-2acf-4858-9e3e-709cb8d68734-data\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.292150 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.292041 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hdn\" (UniqueName: \"kubernetes.io/projected/8f176e3d-2acf-4858-9e3e-709cb8d68734-kube-api-access-f5hdn\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.319479 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.319453 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:12.393295 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.393263 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hdn\" (UniqueName: \"kubernetes.io/projected/8f176e3d-2acf-4858-9e3e-709cb8d68734-kube-api-access-f5hdn\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.393449 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.393428 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f176e3d-2acf-4858-9e3e-709cb8d68734-data\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.393757 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.393740 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f176e3d-2acf-4858-9e3e-709cb8d68734-data\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.401488 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.401459 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hdn\" (UniqueName: \"kubernetes.io/projected/8f176e3d-2acf-4858-9e3e-709cb8d68734-kube-api-access-f5hdn\") pod \"postgres-69fb4594cf-sdm6m\" (UID: \"8f176e3d-2acf-4858-9e3e-709cb8d68734\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.553282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.553251 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:12.690057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:12.690033 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m"] Jul 13 15:01:12.692808 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:01:12.692769 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f176e3d_2acf_4858_9e3e_709cb8d68734.slice/crio-7d2c23a0f6e5985981e730bf66b82e79f3778d1dd4a646eee3f4d6312cfb3541 WatchSource:0}: Error finding container 7d2c23a0f6e5985981e730bf66b82e79f3778d1dd4a646eee3f4d6312cfb3541: Status 404 returned error can't find the container with id 7d2c23a0f6e5985981e730bf66b82e79f3778d1dd4a646eee3f4d6312cfb3541 Jul 13 15:01:13.243859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:13.243821 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" event={"ID":"8f176e3d-2acf-4858-9e3e-709cb8d68734","Type":"ContainerStarted","Data":"7d2c23a0f6e5985981e730bf66b82e79f3778d1dd4a646eee3f4d6312cfb3541"} Jul 13 15:01:23.793598 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:23.793572 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 13 15:01:24.312038 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:24.312001 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" event={"ID":"8f176e3d-2acf-4858-9e3e-709cb8d68734","Type":"ContainerStarted","Data":"90643c9a9ae3ed6e7f5001a2bd445b87778812dab174fb2726da801cef184332"} Jul 13 15:01:24.312207 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:24.312078 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:24.330306 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:24.330260 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" podStartSLOduration=1.233907922 podStartE2EDuration="12.330246191s" totalImagesPullingTime="11.096338269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:24.328259077 +0000 UTC m=+547.218934180" watchObservedRunningTime="2026-07-13 15:01:24.330246191 +0000 UTC m=+547.220921293" Jul 13 15:01:30.346729 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.346701 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdm6m" Jul 13 15:01:30.699174 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.699088 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:30.943539 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.943500 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:30.946151 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.946129 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:30.948666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.948644 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xhtrg\"" Jul 13 15:01:30.955365 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:30.955338 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:31.058372 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.058339 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7pj\" (UniqueName: \"kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj\") pod \"authorino-5d4b67c446-l2k2j\" (UID: \"a43dafd1-d732-4357-bad9-7cb59658109e\") " pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:31.159721 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.159687 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7pj\" (UniqueName: \"kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj\") pod \"authorino-5d4b67c446-l2k2j\" (UID: \"a43dafd1-d732-4357-bad9-7cb59658109e\") " pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:31.171356 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.171329 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7pj\" (UniqueName: \"kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj\") pod \"authorino-5d4b67c446-l2k2j\" (UID: \"a43dafd1-d732-4357-bad9-7cb59658109e\") " pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:31.222021 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.221947 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:31.222211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.222199 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:31.249044 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.249005 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:01:31.252135 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.252115 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.255450 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.255417 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 15:01:31.264449 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.264422 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:01:31.360982 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.360952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsdfq\" (UniqueName: \"kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.361348 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.361072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.361624 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.361604 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:31.363646 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:01:31.363622 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda43dafd1_d732_4357_bad9_7cb59658109e.slice/crio-97aaa87788dd9ddb15068f46290dbef2a235719328701b3b8c595aca142e893f WatchSource:0}: Error finding container 97aaa87788dd9ddb15068f46290dbef2a235719328701b3b8c595aca142e893f: Status 404 returned error can't find the container with id 97aaa87788dd9ddb15068f46290dbef2a235719328701b3b8c595aca142e893f Jul 13 15:01:31.462224 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.462185 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.462436 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.462269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdfq\" (UniqueName: \"kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.464794 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.464761 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.470568 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.470543 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdfq\" (UniqueName: \"kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq\") pod \"authorino-6cf49fd4f9-sjtq5\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.562459 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.562429 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:01:31.695139 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.695114 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:01:31.696765 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:01:31.696738 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad6128fc_4242_4857_80dc_1c3a8fd83989.slice/crio-645930221c86e4d5499a3829983b7625f69609de74c7f084209eb2830dfa07fc WatchSource:0}: Error finding container 645930221c86e4d5499a3829983b7625f69609de74c7f084209eb2830dfa07fc: Status 404 returned error can't find the container with id 645930221c86e4d5499a3829983b7625f69609de74c7f084209eb2830dfa07fc Jul 13 15:01:31.950268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:31.950188 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:32.351854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:32.351819 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" event={"ID":"ad6128fc-4242-4857-80dc-1c3a8fd83989","Type":"ContainerStarted","Data":"645930221c86e4d5499a3829983b7625f69609de74c7f084209eb2830dfa07fc"} Jul 13 15:01:32.352905 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:32.352879 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" event={"ID":"a43dafd1-d732-4357-bad9-7cb59658109e","Type":"ContainerStarted","Data":"97aaa87788dd9ddb15068f46290dbef2a235719328701b3b8c595aca142e893f"} Jul 13 15:01:35.368777 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.368737 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" event={"ID":"ad6128fc-4242-4857-80dc-1c3a8fd83989","Type":"ContainerStarted","Data":"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a"} Jul 13 15:01:35.370215 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.370188 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" event={"ID":"a43dafd1-d732-4357-bad9-7cb59658109e","Type":"ContainerStarted","Data":"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b"} Jul 13 15:01:35.370320 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.370264 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" podUID="a43dafd1-d732-4357-bad9-7cb59658109e" containerName="authorino" containerID="cri-o://b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b" gracePeriod=30 Jul 13 15:01:35.387150 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.387107 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" podStartSLOduration=1.309292981 podStartE2EDuration="4.387084926s" totalImagesPullingTime="3.077791945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:35.384822372 +0000 UTC m=+558.275497473" watchObservedRunningTime="2026-07-13 15:01:35.387084926 +0000 UTC m=+558.277760063" Jul 13 15:01:35.401263 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.401219 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" podStartSLOduration=2.36970194 podStartE2EDuration="5.401203613s" totalImagesPullingTime="3.031501673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:35.400241073 +0000 UTC m=+558.290916176" watchObservedRunningTime="2026-07-13 15:01:35.401203613 +0000 UTC m=+558.291878715" Jul 13 15:01:35.619644 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.619577 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:35.700852 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.700814 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc7pj\" (UniqueName: \"kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj\") pod \"a43dafd1-d732-4357-bad9-7cb59658109e\" (UID: \"a43dafd1-d732-4357-bad9-7cb59658109e\") " Jul 13 15:01:35.702895 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.702864 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj" (OuterVolumeSpecName: "kube-api-access-xc7pj") pod "a43dafd1-d732-4357-bad9-7cb59658109e" (UID: "a43dafd1-d732-4357-bad9-7cb59658109e"). InnerVolumeSpecName "kube-api-access-xc7pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:01:35.802248 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:35.802204 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7pj\" (UniqueName: \"kubernetes.io/projected/a43dafd1-d732-4357-bad9-7cb59658109e-kube-api-access-xc7pj\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:01:36.376569 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.376533 2554 generic.go:356] "Generic (PLEG): container finished" podID="a43dafd1-d732-4357-bad9-7cb59658109e" containerID="b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b" exitCode=0 Jul 13 15:01:36.377020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.376586 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" Jul 13 15:01:36.377020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.376619 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" event={"ID":"a43dafd1-d732-4357-bad9-7cb59658109e","Type":"ContainerDied","Data":"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b"} Jul 13 15:01:36.377020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.376660 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-l2k2j" event={"ID":"a43dafd1-d732-4357-bad9-7cb59658109e","Type":"ContainerDied","Data":"97aaa87788dd9ddb15068f46290dbef2a235719328701b3b8c595aca142e893f"} Jul 13 15:01:36.377020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.376679 2554 scope.go:117] "RemoveContainer" containerID="b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b" Jul 13 15:01:36.385715 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.385701 2554 scope.go:117] "RemoveContainer" containerID="b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b" Jul 13 15:01:36.385982 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:01:36.385966 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b\": container with ID starting with b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b not found: ID does not exist" containerID="b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b" Jul 13 15:01:36.386034 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.385989 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b"} err="failed to get container status \"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b\": rpc error: code = NotFound desc = could not find container \"b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b\": container with ID starting with b32a129115c2fd9ed919ea9fa180946aacc35f3dc7370056f44e2afdb9f8ac9b not found: ID does not exist" Jul 13 15:01:36.414027 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.414001 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:36.419006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:36.418983 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l2k2j"] Jul 13 15:01:37.679169 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:37.679140 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43dafd1-d732-4357-bad9-7cb59658109e" path="/var/lib/kubelet/pods/a43dafd1-d732-4357-bad9-7cb59658109e/volumes" Jul 13 15:01:40.339523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.339494 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:40.391903 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.391869 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:40.392286 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.392272 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43dafd1-d732-4357-bad9-7cb59658109e" containerName="authorino" Jul 13 15:01:40.392335 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.392289 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43dafd1-d732-4357-bad9-7cb59658109e" containerName="authorino" Jul 13 15:01:40.392370 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.392354 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43dafd1-d732-4357-bad9-7cb59658109e" containerName="authorino" Jul 13 15:01:40.395480 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.395462 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.399406 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.399235 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 15:01:40.399606 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.399587 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 15:01:40.402186 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.402025 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dx5xz\"" Jul 13 15:01:40.415761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.415738 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:40.481305 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.481263 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:40.545496 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.545459 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjrp\" (UniqueName: \"kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.545653 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.545612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.558273 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.558239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:40.586547 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.586514 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:01:40.589311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.589293 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.600101 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.600079 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:01:40.646697 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.646671 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjrp\" (UniqueName: \"kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.646895 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.646736 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.649078 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.649057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.655514 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.655497 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjrp\" (UniqueName: \"kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp\") pod \"maas-controller-6b85788bb-z9jzf\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.707910 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.707885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:40.747530 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.747414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.747530 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.747515 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.750403 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.750371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:40.844605 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.844583 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:40.847029 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:01:40.846975 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fad02ad_7873_4e3f_8821_0fa4430dc524.slice/crio-d4d14caa143f6fb8cd222fb653dee4b6be28772fdf6b69d0069740429481cc7f WatchSource:0}: Error finding container d4d14caa143f6fb8cd222fb653dee4b6be28772fdf6b69d0069740429481cc7f: Status 404 returned error can't find the container with id d4d14caa143f6fb8cd222fb653dee4b6be28772fdf6b69d0069740429481cc7f Jul 13 15:01:40.848423 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.848400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.848503 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.848480 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.850832 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.850808 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.858582 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.858562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf\") pod \"maas-controller-6d9c8dd9b7-rghz7\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:40.900743 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:40.900714 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:41.033364 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:41.033340 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:01:41.035012 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:01:41.034981 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d12edb3_17a0_4e13_9ee3_d5d4d346126f.slice/crio-0225ede38632192a506cc728f556d29c0ce4939065a15fc81ab476a77e693c0b WatchSource:0}: Error finding container 0225ede38632192a506cc728f556d29c0ce4939065a15fc81ab476a77e693c0b: Status 404 returned error can't find the container with id 0225ede38632192a506cc728f556d29c0ce4939065a15fc81ab476a77e693c0b Jul 13 15:01:41.403057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:41.403026 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" event={"ID":"9d12edb3-17a0-4e13-9ee3-d5d4d346126f","Type":"ContainerStarted","Data":"0225ede38632192a506cc728f556d29c0ce4939065a15fc81ab476a77e693c0b"} Jul 13 15:01:41.404309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:41.404282 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85788bb-z9jzf" event={"ID":"0fad02ad-7873-4e3f-8821-0fa4430dc524","Type":"ContainerStarted","Data":"d4d14caa143f6fb8cd222fb653dee4b6be28772fdf6b69d0069740429481cc7f"} Jul 13 15:01:44.712219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:44.712183 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:44.809944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:44.809853 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:45.431367 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.431335 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" event={"ID":"9d12edb3-17a0-4e13-9ee3-d5d4d346126f","Type":"ContainerStarted","Data":"7c89727cb25ff909802b7c6aca3cbb3a93c89067bd693fe5607b978d3eab11fe"} Jul 13 15:01:45.431566 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.431422 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:45.432915 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.432895 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85788bb-z9jzf" event={"ID":"0fad02ad-7873-4e3f-8821-0fa4430dc524","Type":"ContainerStarted","Data":"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a"} Jul 13 15:01:45.433031 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.432945 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:45.450590 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.450538 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" podStartSLOduration=1.9566797139999998 podStartE2EDuration="5.450524069s" totalImagesPullingTime="3.493844355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:45.450254252 +0000 UTC m=+568.340929370" watchObservedRunningTime="2026-07-13 15:01:45.450524069 +0000 UTC m=+568.341199170" Jul 13 15:01:45.470432 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:45.470384 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b85788bb-z9jzf" podStartSLOduration=1.775661562 podStartE2EDuration="5.470369153s" totalImagesPullingTime="3.694707591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:45.468160116 +0000 UTC m=+568.358835241" watchObservedRunningTime="2026-07-13 15:01:45.470369153 +0000 UTC m=+568.361044255" Jul 13 15:01:56.440149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.440118 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:56.440607 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.440189 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:01:56.507813 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.507750 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:56.508616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.508037 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b85788bb-z9jzf" podUID="0fad02ad-7873-4e3f-8821-0fa4430dc524" containerName="manager" containerID="cri-o://3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a" gracePeriod=30 Jul 13 15:01:56.761349 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.761325 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:56.789352 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.789316 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert\") pod \"0fad02ad-7873-4e3f-8821-0fa4430dc524\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " Jul 13 15:01:56.789477 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.789394 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tjrp\" (UniqueName: \"kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp\") pod \"0fad02ad-7873-4e3f-8821-0fa4430dc524\" (UID: \"0fad02ad-7873-4e3f-8821-0fa4430dc524\") " Jul 13 15:01:56.791475 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.791448 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp" (OuterVolumeSpecName: "kube-api-access-9tjrp") pod "0fad02ad-7873-4e3f-8821-0fa4430dc524" (UID: "0fad02ad-7873-4e3f-8821-0fa4430dc524"). InnerVolumeSpecName "kube-api-access-9tjrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:01:56.791475 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.791454 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0fad02ad-7873-4e3f-8821-0fa4430dc524" (UID: "0fad02ad-7873-4e3f-8821-0fa4430dc524"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:01:56.890189 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.890156 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fad02ad-7873-4e3f-8821-0fa4430dc524-webhook-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:01:56.890189 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:56.890180 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjrp\" (UniqueName: \"kubernetes.io/projected/0fad02ad-7873-4e3f-8821-0fa4430dc524-kube-api-access-9tjrp\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:01:57.493707 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.493661 2554 generic.go:356] "Generic (PLEG): container finished" podID="0fad02ad-7873-4e3f-8821-0fa4430dc524" containerID="3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a" exitCode=0 Jul 13 15:01:57.494211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.493735 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85788bb-z9jzf" event={"ID":"0fad02ad-7873-4e3f-8821-0fa4430dc524","Type":"ContainerDied","Data":"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a"} Jul 13 15:01:57.494211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.493857 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b85788bb-z9jzf" event={"ID":"0fad02ad-7873-4e3f-8821-0fa4430dc524","Type":"ContainerDied","Data":"d4d14caa143f6fb8cd222fb653dee4b6be28772fdf6b69d0069740429481cc7f"} Jul 13 15:01:57.494211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.493898 2554 scope.go:117] "RemoveContainer" containerID="3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a" Jul 13 15:01:57.494211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.494095 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b85788bb-z9jzf" Jul 13 15:01:57.508737 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.508717 2554 scope.go:117] "RemoveContainer" containerID="3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a" Jul 13 15:01:57.509016 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:01:57.508994 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a\": container with ID starting with 3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a not found: ID does not exist" containerID="3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a" Jul 13 15:01:57.509102 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.509024 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a"} err="failed to get container status \"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a\": rpc error: code = NotFound desc = could not find container \"3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a\": container with ID starting with 3cbb891165b7ddfbdbf0678000169e1a22842744cf6b275a5bc6bd8f20975f4a not found: ID does not exist" Jul 13 15:01:57.519941 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.519919 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:57.523523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.523502 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b85788bb-z9jzf"] Jul 13 15:01:57.679153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:01:57.679124 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fad02ad-7873-4e3f-8821-0fa4430dc524" path="/var/lib/kubelet/pods/0fad02ad-7873-4e3f-8821-0fa4430dc524/volumes" Jul 13 15:02:01.042503 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:01.042470 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:01.105701 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:01.105666 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:17.268231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.268194 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:17.366750 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.366718 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:17.562152 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.562066 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:02:17.562323 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.562308 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:02:17.568156 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.568133 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:02:17.568275 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.568214 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:02:17.842746 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.842664 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z"] Jul 13 15:02:17.843088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.843073 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fad02ad-7873-4e3f-8821-0fa4430dc524" containerName="manager" Jul 13 15:02:17.843149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.843089 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fad02ad-7873-4e3f-8821-0fa4430dc524" containerName="manager" Jul 13 15:02:17.843201 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.843160 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fad02ad-7873-4e3f-8821-0fa4430dc524" containerName="manager" Jul 13 15:02:17.847866 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.847849 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:17.850689 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.850666 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 15:02:17.850821 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.850688 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\"" Jul 13 15:02:17.856941 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.856920 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z"] Jul 13 15:02:17.975429 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.975401 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bc57b3-2783-45a4-ad0c-48c314b81b1a-maas-api-tls\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:17.975429 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.975434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc57b3-2783-45a4-ad0c-48c314b81b1a-openshift-service-ca\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:17.975636 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:17.975457 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrc9k\" (UniqueName: \"kubernetes.io/projected/25bc57b3-2783-45a4-ad0c-48c314b81b1a-kube-api-access-nrc9k\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.076683 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.076646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bc57b3-2783-45a4-ad0c-48c314b81b1a-maas-api-tls\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.076683 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.076687 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc57b3-2783-45a4-ad0c-48c314b81b1a-openshift-service-ca\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.077006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.076824 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrc9k\" (UniqueName: \"kubernetes.io/projected/25bc57b3-2783-45a4-ad0c-48c314b81b1a-kube-api-access-nrc9k\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.077442 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.077424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc57b3-2783-45a4-ad0c-48c314b81b1a-openshift-service-ca\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.079413 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.079387 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25bc57b3-2783-45a4-ad0c-48c314b81b1a-maas-api-tls\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.089107 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.089084 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrc9k\" (UniqueName: \"kubernetes.io/projected/25bc57b3-2783-45a4-ad0c-48c314b81b1a-kube-api-access-nrc9k\") pod \"maas-api-86d5b84cf-thz6z\" (UID: \"25bc57b3-2783-45a4-ad0c-48c314b81b1a\") " pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.159239 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.159151 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:18.303740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.303645 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z"] Jul 13 15:02:18.317946 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:02:18.306825 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bc57b3_2783_45a4_ad0c_48c314b81b1a.slice/crio-297361574d4357a6d8e9ac4726e33c5c35f4b3841a4ec1e59dde88286bbe82a6 WatchSource:0}: Error finding container 297361574d4357a6d8e9ac4726e33c5c35f4b3841a4ec1e59dde88286bbe82a6: Status 404 returned error can't find the container with id 297361574d4357a6d8e9ac4726e33c5c35f4b3841a4ec1e59dde88286bbe82a6 Jul 13 15:02:18.411267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.411186 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt"] Jul 13 15:02:18.414620 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.414602 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.417344 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.417324 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 15:02:18.427361 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.427341 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt"] Jul 13 15:02:18.432722 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.432698 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-v5f85"] Jul 13 15:02:18.436206 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.436187 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.439758 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.439737 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lcgdb\"" Jul 13 15:02:18.447557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.447536 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-v5f85"] Jul 13 15:02:18.481082 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.481036 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea2ef8e-4abe-4918-94df-6e7d3e23f492-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.481268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.481154 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhfh\" (UniqueName: \"kubernetes.io/projected/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-kube-api-access-cwhfh\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.481268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.481191 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/aea2ef8e-4abe-4918-94df-6e7d3e23f492-kube-api-access-bxmq8\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.481268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.481240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.582523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.582488 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhfh\" (UniqueName: \"kubernetes.io/projected/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-kube-api-access-cwhfh\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.582695 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.582532 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/aea2ef8e-4abe-4918-94df-6e7d3e23f492-kube-api-access-bxmq8\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.582695 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.582570 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.582695 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.582594 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea2ef8e-4abe-4918-94df-6e7d3e23f492-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.583396 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.583376 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.583444 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.583376 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea2ef8e-4abe-4918-94df-6e7d3e23f492-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.592083 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.592050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmq8\" (UniqueName: \"kubernetes.io/projected/aea2ef8e-4abe-4918-94df-6e7d3e23f492-kube-api-access-bxmq8\") pod \"payload-pre-processing-74584bb9f8-bpgmt\" (UID: \"aea2ef8e-4abe-4918-94df-6e7d3e23f492\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.592210 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.592193 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhfh\" (UniqueName: \"kubernetes.io/projected/622f4a8e-9934-4543-90b4-bec1dd3ce4a0-kube-api-access-cwhfh\") pod \"payload-processing-f9b8d9b9b-v5f85\" (UID: \"622f4a8e-9934-4543-90b4-bec1dd3ce4a0\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.600229 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.600191 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" event={"ID":"25bc57b3-2783-45a4-ad0c-48c314b81b1a","Type":"ContainerStarted","Data":"297361574d4357a6d8e9ac4726e33c5c35f4b3841a4ec1e59dde88286bbe82a6"} Jul 13 15:02:18.727988 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.727901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:18.748260 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.748237 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:18.765447 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.765415 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:18.865529 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.865470 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:18.882029 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.882001 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt"] Jul 13 15:02:18.884176 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:02:18.884139 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea2ef8e_4abe_4918_94df_6e7d3e23f492.slice/crio-cfd7d201108774708dce8f8c790e778ea0eae8bdef8c2d2154b85930fd1215db WatchSource:0}: Error finding container cfd7d201108774708dce8f8c790e778ea0eae8bdef8c2d2154b85930fd1215db: Status 404 returned error can't find the container with id cfd7d201108774708dce8f8c790e778ea0eae8bdef8c2d2154b85930fd1215db Jul 13 15:02:18.913850 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:18.913829 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-v5f85"] Jul 13 15:02:18.915227 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:02:18.915200 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622f4a8e_9934_4543_90b4_bec1dd3ce4a0.slice/crio-370368c3eea196f5e58482f1e440e2eea5c944c02f464bbf0ff22536d2a08ad8 WatchSource:0}: Error finding container 370368c3eea196f5e58482f1e440e2eea5c944c02f464bbf0ff22536d2a08ad8: Status 404 returned error can't find the container with id 370368c3eea196f5e58482f1e440e2eea5c944c02f464bbf0ff22536d2a08ad8 Jul 13 15:02:19.606477 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:19.606440 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" event={"ID":"aea2ef8e-4abe-4918-94df-6e7d3e23f492","Type":"ContainerStarted","Data":"cfd7d201108774708dce8f8c790e778ea0eae8bdef8c2d2154b85930fd1215db"} Jul 13 15:02:19.607540 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:19.607515 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" event={"ID":"622f4a8e-9934-4543-90b4-bec1dd3ce4a0","Type":"ContainerStarted","Data":"370368c3eea196f5e58482f1e440e2eea5c944c02f464bbf0ff22536d2a08ad8"} Jul 13 15:02:20.617240 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:20.617165 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" event={"ID":"25bc57b3-2783-45a4-ad0c-48c314b81b1a","Type":"ContainerStarted","Data":"c9c16d1c5e19ecf4143bf41e2ceb7b977046db1b286f484be8d5fc74d68b3b16"} Jul 13 15:02:20.617820 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:20.617763 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:20.648038 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:20.647978 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" podStartSLOduration=1.934933322 podStartE2EDuration="3.64795678s" totalImagesPullingTime="1.713023458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:20.647089213 +0000 UTC m=+603.537764328" watchObservedRunningTime="2026-07-13 15:02:20.64795678 +0000 UTC m=+603.538631884" Jul 13 15:02:21.626481 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.626363 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" event={"ID":"aea2ef8e-4abe-4918-94df-6e7d3e23f492","Type":"ContainerStarted","Data":"1ed230706c4e55589cfce904a70bc3e19bd4d2e14462e682628e6471a036ceb9"} Jul 13 15:02:21.627425 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.627398 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:21.630433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.630399 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" event={"ID":"622f4a8e-9934-4543-90b4-bec1dd3ce4a0","Type":"ContainerStarted","Data":"2e17372de9eecddb62f94a9c7340f7448245dbea4031459d6075ccc446d76b42"} Jul 13 15:02:21.630568 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.630453 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:21.645594 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.645528 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" podStartSLOduration=1.147707413 podStartE2EDuration="3.645507486s" totalImagesPullingTime="2.497800073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:21.64338196 +0000 UTC m=+604.534057064" watchObservedRunningTime="2026-07-13 15:02:21.645507486 +0000 UTC m=+604.536182592" Jul 13 15:02:21.661718 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:21.661652 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" podStartSLOduration=1.198489423 podStartE2EDuration="3.661629704s" totalImagesPullingTime="2.463140281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:21.659157086 +0000 UTC m=+604.549832191" watchObservedRunningTime="2026-07-13 15:02:21.661629704 +0000 UTC m=+604.552304818" Jul 13 15:02:26.093978 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:26.093937 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 15:02:26.094365 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:26.094011 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 15:02:27.643467 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.643441 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d5b84cf-thz6z" Jul 13 15:02:27.669825 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.669765 2554 generic.go:356] "Generic (PLEG): container finished" podID="bece848d-6294-46cc-b9da-d2e5f1436678" containerID="1c7643ebd9cc97614347bd624f37ea2a763a98476262992c4c7081333072e202" exitCode=0 Jul 13 15:02:27.670008 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.669833 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" event={"ID":"bece848d-6294-46cc-b9da-d2e5f1436678","Type":"ContainerDied","Data":"1c7643ebd9cc97614347bd624f37ea2a763a98476262992c4c7081333072e202"} Jul 13 15:02:27.670376 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.670357 2554 scope.go:117] "RemoveContainer" containerID="1c7643ebd9cc97614347bd624f37ea2a763a98476262992c4c7081333072e202" Jul 13 15:02:27.671416 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.671382 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:02:27.671523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.671471 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:02:27.671523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:27.671513 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:02:28.687569 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.687530 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" event={"ID":"bece848d-6294-46cc-b9da-d2e5f1436678","Type":"ContainerStarted","Data":"c37ec01db1eb85566083eed408575bd78e21e47f1f7a43e6f8256fccf1835a6e"} Jul 13 15:02:28.714599 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.714565 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:02:28.714762 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.714690 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 13 15:02:28.714762 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.714744 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 13 15:02:28.787594 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.787554 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:28.950608 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:28.950498 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:29.714728 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:29.714691 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 13 15:02:29.715130 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:29.714745 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 13 15:02:30.715004 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:30.714969 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 13 15:02:30.715451 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:30.715026 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 13 15:02:31.714944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:31.714909 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 13 15:02:31.715153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:31.714964 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 13 15:02:32.643004 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:32.642973 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-v5f85" Jul 13 15:02:32.715152 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:32.715125 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wggx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 13 15:02:32.715603 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:32.715178 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" podUID="bece848d-6294-46cc-b9da-d2e5f1436678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 13 15:02:33.648589 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:33.648550 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" podUID="aea2ef8e-4abe-4918-94df-6e7d3e23f492" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.56:9004: connect: connection refused" Jul 13 15:02:33.718763 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:33.718732 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:02:33.719194 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:33.719139 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:02:33.719926 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:33.719906 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wggx2" Jul 13 15:02:43.110528 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:43.110435 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:43.646839 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:43.646781 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-bpgmt" Jul 13 15:02:47.530474 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:47.530439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:48.983392 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.983352 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm"] Jul 13 15:02:48.986497 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.986473 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:48.990865 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.990821 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ttcck\"" Jul 13 15:02:48.990865 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.990837 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 13 15:02:48.990865 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.990860 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 15:02:48.991116 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.990913 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 15:02:48.998709 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:48.998091 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm"] Jul 13 15:02:49.062618 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.062780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ggnr\" (UniqueName: \"kubernetes.io/projected/9430f0cd-ac95-422b-8863-04028ea11020-kube-api-access-8ggnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.062780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.062780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062736 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.062973 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062835 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430f0cd-ac95-422b-8863-04028ea11020-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.062973 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.062887 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164067 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164029 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ggnr\" (UniqueName: \"kubernetes.io/projected/9430f0cd-ac95-422b-8863-04028ea11020-kube-api-access-8ggnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164107 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164132 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164162 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430f0cd-ac95-422b-8863-04028ea11020-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164197 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164619 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164590 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164669 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.164745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.164700 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.166585 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.166560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430f0cd-ac95-422b-8863-04028ea11020-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.166720 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.166705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430f0cd-ac95-422b-8863-04028ea11020-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.171973 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.171956 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ggnr\" (UniqueName: \"kubernetes.io/projected/9430f0cd-ac95-422b-8863-04028ea11020-kube-api-access-8ggnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jphqm\" (UID: \"9430f0cd-ac95-422b-8863-04028ea11020\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.300284 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.300254 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:49.438768 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.438743 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm"] Jul 13 15:02:49.440725 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:02:49.440694 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9430f0cd_ac95_422b_8863_04028ea11020.slice/crio-06aa64947d2ecab179065b624bc08dfbee90122b17121eb6ec9ebd51a2dce249 WatchSource:0}: Error finding container 06aa64947d2ecab179065b624bc08dfbee90122b17121eb6ec9ebd51a2dce249: Status 404 returned error can't find the container with id 06aa64947d2ecab179065b624bc08dfbee90122b17121eb6ec9ebd51a2dce249 Jul 13 15:02:49.442072 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.442051 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 15:02:49.794554 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:49.794520 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" event={"ID":"9430f0cd-ac95-422b-8863-04028ea11020","Type":"ContainerStarted","Data":"06aa64947d2ecab179065b624bc08dfbee90122b17121eb6ec9ebd51a2dce249"} Jul 13 15:02:50.500672 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:50.500629 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:53.816849 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:53.816807 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" event={"ID":"9430f0cd-ac95-422b-8863-04028ea11020","Type":"ContainerStarted","Data":"205513364bf6a30a21177d8e96ef4b99393d7bb1a6e70df9f96fc2a8fe0f1f8e"} Jul 13 15:02:53.836990 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:53.836939 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" podStartSLOduration=1.782292046 podStartE2EDuration="5.836922823s" totalImagesPullingTime="4.054630777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:53.834761853 +0000 UTC m=+636.725436955" watchObservedRunningTime="2026-07-13 15:02:53.836922823 +0000 UTC m=+636.727597925" Jul 13 15:02:58.277115 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.277080 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw"] Jul 13 15:02:58.286543 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.286507 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.289196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.289172 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 13 15:02:58.289727 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.289708 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw"] Jul 13 15:02:58.346335 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346302 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.346335 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346336 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7q2\" (UniqueName: \"kubernetes.io/projected/5653cfd3-94ff-41ef-9818-d07f2e721451-kube-api-access-gd7q2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.346550 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346363 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.346550 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346381 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.346550 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346463 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.346663 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.346548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5653cfd3-94ff-41ef-9818-d07f2e721451-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.389528 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.389481 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:58.447835 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.447806 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.447843 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7q2\" (UniqueName: \"kubernetes.io/projected/5653cfd3-94ff-41ef-9818-d07f2e721451-kube-api-access-gd7q2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.447870 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.447886 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.447907 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448209 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.448035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5653cfd3-94ff-41ef-9818-d07f2e721451-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.448255 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448328 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.448295 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.448379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.448338 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.450195 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.450171 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5653cfd3-94ff-41ef-9818-d07f2e721451-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.450463 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.450445 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5653cfd3-94ff-41ef-9818-d07f2e721451-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.456638 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.456615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7q2\" (UniqueName: \"kubernetes.io/projected/5653cfd3-94ff-41ef-9818-d07f2e721451-kube-api-access-gd7q2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k8cnw\" (UID: \"5653cfd3-94ff-41ef-9818-d07f2e721451\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.598677 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.598588 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:02:58.737390 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.737363 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw"] Jul 13 15:02:58.739243 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:02:58.739213 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5653cfd3_94ff_41ef_9818_d07f2e721451.slice/crio-0a4769cc7f447627e5929788ac0117b78fb86d1c6eca67e4014c3c5748c41454 WatchSource:0}: Error finding container 0a4769cc7f447627e5929788ac0117b78fb86d1c6eca67e4014c3c5748c41454: Status 404 returned error can't find the container with id 0a4769cc7f447627e5929788ac0117b78fb86d1c6eca67e4014c3c5748c41454 Jul 13 15:02:58.843277 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:58.843243 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" event={"ID":"5653cfd3-94ff-41ef-9818-d07f2e721451","Type":"ContainerStarted","Data":"0a4769cc7f447627e5929788ac0117b78fb86d1c6eca67e4014c3c5748c41454"} Jul 13 15:02:59.301392 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.301359 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:59.301828 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.301414 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:59.314023 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.313997 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:59.850042 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.850009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" event={"ID":"5653cfd3-94ff-41ef-9818-d07f2e721451","Type":"ContainerStarted","Data":"15aec30d441c099f0361d9fc46b801b406820241c4b3db43f7db1a57f859d6b9"} Jul 13 15:02:59.862696 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.862668 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jphqm" Jul 13 15:02:59.871665 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:02:59.871616 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" podStartSLOduration=1.638370646 podStartE2EDuration="1.871601303s" totalImagesPullingTime="233.230657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:59.868761228 +0000 UTC m=+642.759436331" watchObservedRunningTime="2026-07-13 15:02:59.871601303 +0000 UTC m=+642.762276404" Jul 13 15:03:08.599118 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:08.599082 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:03:08.599747 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:08.599138 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:03:08.611489 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:08.611464 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:03:08.914322 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:08.914237 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k8cnw" Jul 13 15:03:09.010575 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:09.010523 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:42.857234 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:42.857197 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:44.298448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:44.298406 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:59.472006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:59.471970 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:59.557003 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:03:59.556973 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:10.437402 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:04:10.437306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:47.979038 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:04:47.978997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:48.041249 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:04:48.041217 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:05:20.883877 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:20.883834 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:05:20.984705 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:20.984666 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:05:21.623318 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.623281 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:05:21.623578 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.623515 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" podUID="ad6128fc-4242-4857-80dc-1c3a8fd83989" containerName="authorino" containerID="cri-o://cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a" gracePeriod=30 Jul 13 15:05:21.886975 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.886914 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:05:21.981000 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.980965 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert\") pod \"ad6128fc-4242-4857-80dc-1c3a8fd83989\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " Jul 13 15:05:21.981171 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.981026 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsdfq\" (UniqueName: \"kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq\") pod \"ad6128fc-4242-4857-80dc-1c3a8fd83989\" (UID: \"ad6128fc-4242-4857-80dc-1c3a8fd83989\") " Jul 13 15:05:21.983139 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.983108 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq" (OuterVolumeSpecName: "kube-api-access-wsdfq") pod "ad6128fc-4242-4857-80dc-1c3a8fd83989" (UID: "ad6128fc-4242-4857-80dc-1c3a8fd83989"). InnerVolumeSpecName "kube-api-access-wsdfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:05:21.992441 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:21.992409 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ad6128fc-4242-4857-80dc-1c3a8fd83989" (UID: "ad6128fc-4242-4857-80dc-1c3a8fd83989"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:05:22.082334 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.082298 2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad6128fc-4242-4857-80dc-1c3a8fd83989-tls-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:05:22.082334 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.082329 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsdfq\" (UniqueName: \"kubernetes.io/projected/ad6128fc-4242-4857-80dc-1c3a8fd83989-kube-api-access-wsdfq\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:05:22.149250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.149125 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:05:22.312458 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.312420 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:05:22.595129 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.595098 2554 generic.go:356] "Generic (PLEG): container finished" podID="ad6128fc-4242-4857-80dc-1c3a8fd83989" containerID="cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a" exitCode=0 Jul 13 15:05:22.595309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.595141 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" Jul 13 15:05:22.595309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.595177 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" event={"ID":"ad6128fc-4242-4857-80dc-1c3a8fd83989","Type":"ContainerDied","Data":"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a"} Jul 13 15:05:22.595309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.595219 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf49fd4f9-sjtq5" event={"ID":"ad6128fc-4242-4857-80dc-1c3a8fd83989","Type":"ContainerDied","Data":"645930221c86e4d5499a3829983b7625f69609de74c7f084209eb2830dfa07fc"} Jul 13 15:05:22.595309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.595236 2554 scope.go:117] "RemoveContainer" containerID="cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a" Jul 13 15:05:22.604517 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.604501 2554 scope.go:117] "RemoveContainer" containerID="cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a" Jul 13 15:05:22.604765 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:05:22.604748 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a\": container with ID starting with cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a not found: ID does not exist" containerID="cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a" Jul 13 15:05:22.604844 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.604774 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a"} err="failed to get container status \"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a\": rpc error: code = NotFound desc = could not find container \"cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a\": container with ID starting with cb8f84a39c5e0abf6cc5195a1d36470f755cf65839863d33761ce48d9d4e9e9a not found: ID does not exist" Jul 13 15:05:22.617553 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.617526 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:05:22.621921 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:22.621897 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cf49fd4f9-sjtq5"] Jul 13 15:05:23.679277 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:23.679243 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad6128fc-4242-4857-80dc-1c3a8fd83989" path="/var/lib/kubelet/pods/ad6128fc-4242-4857-80dc-1c3a8fd83989/volumes" Jul 13 15:05:29.485843 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:05:29.485775 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:06:39.760165 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:39.760127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:06:39.853944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:39.853911 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:06:39.854182 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:39.854115 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" podUID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" containerName="manager" containerID="cri-o://7c89727cb25ff909802b7c6aca3cbb3a93c89067bd693fe5607b978d3eab11fe" gracePeriod=30 Jul 13 15:06:40.006241 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.006209 2554 generic.go:356] "Generic (PLEG): container finished" podID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" containerID="7c89727cb25ff909802b7c6aca3cbb3a93c89067bd693fe5607b978d3eab11fe" exitCode=0 Jul 13 15:06:40.006416 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.006260 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" event={"ID":"9d12edb3-17a0-4e13-9ee3-d5d4d346126f","Type":"ContainerDied","Data":"7c89727cb25ff909802b7c6aca3cbb3a93c89067bd693fe5607b978d3eab11fe"} Jul 13 15:06:40.111649 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.111627 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:06:40.269979 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.269946 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf\") pod \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " Jul 13 15:06:40.270180 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.270017 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert\") pod \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\" (UID: \"9d12edb3-17a0-4e13-9ee3-d5d4d346126f\") " Jul 13 15:06:40.272176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.272143 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9d12edb3-17a0-4e13-9ee3-d5d4d346126f" (UID: "9d12edb3-17a0-4e13-9ee3-d5d4d346126f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:06:40.272176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.272160 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf" (OuterVolumeSpecName: "kube-api-access-wfmwf") pod "9d12edb3-17a0-4e13-9ee3-d5d4d346126f" (UID: "9d12edb3-17a0-4e13-9ee3-d5d4d346126f"). InnerVolumeSpecName "kube-api-access-wfmwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:06:40.370960 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.370927 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-kube-api-access-wfmwf\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:06:40.370960 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:40.370955 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d12edb3-17a0-4e13-9ee3-d5d4d346126f-webhook-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:06:41.012305 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.012273 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" Jul 13 15:06:41.012305 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.012282 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-rghz7" event={"ID":"9d12edb3-17a0-4e13-9ee3-d5d4d346126f","Type":"ContainerDied","Data":"0225ede38632192a506cc728f556d29c0ce4939065a15fc81ab476a77e693c0b"} Jul 13 15:06:41.012800 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.012346 2554 scope.go:117] "RemoveContainer" containerID="7c89727cb25ff909802b7c6aca3cbb3a93c89067bd693fe5607b978d3eab11fe" Jul 13 15:06:41.038929 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.038871 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:06:41.041107 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.041086 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-rghz7"] Jul 13 15:06:41.679798 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.679750 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" path="/var/lib/kubelet/pods/9d12edb3-17a0-4e13-9ee3-d5d4d346126f/volumes" Jul 13 15:06:41.831779 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.831740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:06:41.998686 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.998596 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-dkrtr"] Jul 13 15:06:41.999115 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999098 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" containerName="manager" Jul 13 15:06:41.999211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999118 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" containerName="manager" Jul 13 15:06:41.999211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999136 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6128fc-4242-4857-80dc-1c3a8fd83989" containerName="authorino" Jul 13 15:06:41.999211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999144 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6128fc-4242-4857-80dc-1c3a8fd83989" containerName="authorino" Jul 13 15:06:41.999357 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999246 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d12edb3-17a0-4e13-9ee3-d5d4d346126f" containerName="manager" Jul 13 15:06:41.999357 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:41.999264 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6128fc-4242-4857-80dc-1c3a8fd83989" containerName="authorino" Jul 13 15:06:42.001635 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.001615 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.005567 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.005546 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dx5xz\"" Jul 13 15:06:42.005679 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.005552 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 15:06:42.005679 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.005577 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 15:06:42.016064 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.016040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-dkrtr"] Jul 13 15:06:42.087458 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.087427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc31e8d0-a137-4fac-9233-99094be96b31-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.087618 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.087485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grhwv\" (UniqueName: \"kubernetes.io/projected/bc31e8d0-a137-4fac-9233-99094be96b31-kube-api-access-grhwv\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.188977 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.188942 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grhwv\" (UniqueName: \"kubernetes.io/projected/bc31e8d0-a137-4fac-9233-99094be96b31-kube-api-access-grhwv\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.189154 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.189072 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc31e8d0-a137-4fac-9233-99094be96b31-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.191649 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.191624 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc31e8d0-a137-4fac-9233-99094be96b31-webhook-cert\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.199344 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.199317 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grhwv\" (UniqueName: \"kubernetes.io/projected/bc31e8d0-a137-4fac-9233-99094be96b31-kube-api-access-grhwv\") pod \"maas-controller-6d9c8dd9b7-dkrtr\" (UID: \"bc31e8d0-a137-4fac-9233-99094be96b31\") " pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.312344 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.312311 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:42.463942 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:42.463914 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9c8dd9b7-dkrtr"] Jul 13 15:06:42.464133 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:06:42.464111 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc31e8d0_a137_4fac_9233_99094be96b31.slice/crio-636b1d0dad812a259b274e9c4d609a3b3f9fed6e8d000f83af05770a57ed062f WatchSource:0}: Error finding container 636b1d0dad812a259b274e9c4d609a3b3f9fed6e8d000f83af05770a57ed062f: Status 404 returned error can't find the container with id 636b1d0dad812a259b274e9c4d609a3b3f9fed6e8d000f83af05770a57ed062f Jul 13 15:06:43.025289 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:43.025254 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" event={"ID":"bc31e8d0-a137-4fac-9233-99094be96b31","Type":"ContainerStarted","Data":"636b1d0dad812a259b274e9c4d609a3b3f9fed6e8d000f83af05770a57ed062f"} Jul 13 15:06:44.032276 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:44.032236 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" event={"ID":"bc31e8d0-a137-4fac-9233-99094be96b31","Type":"ContainerStarted","Data":"91e9cc0a1dcfd9fa6afb4bc44c9a1ca1c1d5179e782de72d503d11ee47d7e2cf"} Jul 13 15:06:44.032670 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:44.032305 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:44.055112 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:44.055062 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" podStartSLOduration=1.819151935 podStartE2EDuration="3.055047423s" totalImagesPullingTime="1.235895488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:06:44.05180686 +0000 UTC m=+866.942481952" watchObservedRunningTime="2026-07-13 15:06:44.055047423 +0000 UTC m=+866.945722524" Jul 13 15:06:55.039165 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:55.039135 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9c8dd9b7-dkrtr" Jul 13 15:06:56.509437 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:06:56.509400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:17.597929 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:17.597863 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:07:17.601529 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:17.601506 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:07:17.604405 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:17.604388 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:07:17.608223 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:17.608206 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:07:22.302764 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:22.302712 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:22.487537 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:22.487496 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:22.585446 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:22.585358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:28.856497 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:28.856460 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:28.967961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:28.967930 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:30.178512 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:30.178478 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:30.279818 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:30.279761 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:35.649646 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:35.649610 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:07:35.742615 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:07:35.742579 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:08:34.312848 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:08:34.312814 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:10:50.628656 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:10:50.628621 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:11:04.023730 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:04.023694 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:11:37.328080 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:37.328040 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:11:37.328625 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:37.328271 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" podUID="a352df7f-ad62-427d-9531-c7c1c0febf22" containerName="manager" containerID="cri-o://5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e" gracePeriod=10 Jul 13 15:11:40.382338 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.382314 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:11:40.489911 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.489832 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume\") pod \"a352df7f-ad62-427d-9531-c7c1c0febf22\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " Jul 13 15:11:40.489911 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.489905 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjnzw\" (UniqueName: \"kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw\") pod \"a352df7f-ad62-427d-9531-c7c1c0febf22\" (UID: \"a352df7f-ad62-427d-9531-c7c1c0febf22\") " Jul 13 15:11:40.490230 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.490199 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a352df7f-ad62-427d-9531-c7c1c0febf22" (UID: "a352df7f-ad62-427d-9531-c7c1c0febf22"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:11:40.491857 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.491835 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw" (OuterVolumeSpecName: "kube-api-access-cjnzw") pod "a352df7f-ad62-427d-9531-c7c1c0febf22" (UID: "a352df7f-ad62-427d-9531-c7c1c0febf22"). InnerVolumeSpecName "kube-api-access-cjnzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:11:40.576502 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.576470 2554 generic.go:356] "Generic (PLEG): container finished" podID="a352df7f-ad62-427d-9531-c7c1c0febf22" containerID="5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e" exitCode=0 Jul 13 15:11:40.576666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.576522 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" Jul 13 15:11:40.576666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.576553 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" event={"ID":"a352df7f-ad62-427d-9531-c7c1c0febf22","Type":"ContainerDied","Data":"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e"} Jul 13 15:11:40.576666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.576601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526" event={"ID":"a352df7f-ad62-427d-9531-c7c1c0febf22","Type":"ContainerDied","Data":"566232b588884a5b8fb848d5f3f0735cdc305b4c4efd5f38396bcbd9a2134dfd"} Jul 13 15:11:40.576666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.576621 2554 scope.go:117] "RemoveContainer" containerID="5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e" Jul 13 15:11:40.586640 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.586620 2554 scope.go:117] "RemoveContainer" containerID="5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e" Jul 13 15:11:40.586893 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:11:40.586875 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e\": container with ID starting with 5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e not found: ID does not exist" containerID="5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e" Jul 13 15:11:40.586963 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.586901 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e"} err="failed to get container status \"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e\": rpc error: code = NotFound desc = could not find container \"5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e\": container with ID starting with 5fbb2d5802d23da9fc8c01e5a2207f1f7b3a040599ba09910471d438af07fb2e not found: ID does not exist" Jul 13 15:11:40.591051 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.591033 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a352df7f-ad62-427d-9531-c7c1c0febf22-extensions-socket-volume\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:11:40.591123 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.591052 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnzw\" (UniqueName: \"kubernetes.io/projected/a352df7f-ad62-427d-9531-c7c1c0febf22-kube-api-access-cjnzw\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:11:40.600300 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.600279 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:11:40.605254 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:40.605233 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d5526"] Jul 13 15:11:41.679323 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:11:41.679289 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a352df7f-ad62-427d-9531-c7c1c0febf22" path="/var/lib/kubelet/pods/a352df7f-ad62-427d-9531-c7c1c0febf22/volumes" Jul 13 15:12:17.632999 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:17.632971 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:12:17.640036 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:17.640012 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:12:17.641433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:17.641411 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:12:17.654630 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:17.654608 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:12:38.604923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.604891 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx"] Jul 13 15:12:38.607195 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.605316 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a352df7f-ad62-427d-9531-c7c1c0febf22" containerName="manager" Jul 13 15:12:38.607195 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.605327 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a352df7f-ad62-427d-9531-c7c1c0febf22" containerName="manager" Jul 13 15:12:38.607195 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.605400 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a352df7f-ad62-427d-9531-c7c1c0febf22" containerName="manager" Jul 13 15:12:38.608201 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.608185 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.611986 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.611962 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-76kpm\"" Jul 13 15:12:38.621755 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.621733 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx"] Jul 13 15:12:38.716956 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.716926 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjhf\" (UniqueName: \"kubernetes.io/projected/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-kube-api-access-nhjhf\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.717140 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.716970 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.818436 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.818403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjhf\" (UniqueName: \"kubernetes.io/projected/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-kube-api-access-nhjhf\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.818626 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.818465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.818897 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.818869 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.827653 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.827623 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjhf\" (UniqueName: \"kubernetes.io/projected/b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1-kube-api-access-nhjhf\") pod \"kuadrant-operator-controller-manager-86878c769f-xg5cx\" (UID: \"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:38.919817 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:38.919732 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:39.065360 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:39.065329 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx"] Jul 13 15:12:39.067136 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:12:39.067109 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f092ed_5c3f_4ed2_90eb_f397ad18f7a1.slice/crio-276fc5a75facf4754a15c351a0fdf7be2f4491070b786ffc59ab6e8d2a72d4c6 WatchSource:0}: Error finding container 276fc5a75facf4754a15c351a0fdf7be2f4491070b786ffc59ab6e8d2a72d4c6: Status 404 returned error can't find the container with id 276fc5a75facf4754a15c351a0fdf7be2f4491070b786ffc59ab6e8d2a72d4c6 Jul 13 15:12:39.868709 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:39.868673 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" event={"ID":"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1","Type":"ContainerStarted","Data":"9d9f3c0deacaa3f3dea96bbfe40f8e993b216d473a6863017a23b6943a289ffa"} Jul 13 15:12:39.869227 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:39.868733 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" event={"ID":"b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1","Type":"ContainerStarted","Data":"276fc5a75facf4754a15c351a0fdf7be2f4491070b786ffc59ab6e8d2a72d4c6"} Jul 13 15:12:39.869227 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:39.868762 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:12:39.893489 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:39.893434 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" podStartSLOduration=1.893416968 podStartE2EDuration="1.893416968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:12:39.890812447 +0000 UTC m=+1222.781487541" watchObservedRunningTime="2026-07-13 15:12:39.893416968 +0000 UTC m=+1222.784092134" Jul 13 15:12:50.876096 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:12:50.876059 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xg5cx" Jul 13 15:13:05.716379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:13:05.716340 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:15:00.154284 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.154242 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc"] Jul 13 15:15:00.158111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.158090 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" Jul 13 15:15:00.161255 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.161225 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:15:00.167750 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.167724 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" Jul 13 15:15:00.176045 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.176014 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc"] Jul 13 15:15:00.307918 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.307889 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc"] Jul 13 15:15:00.312750 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:15:00.312714 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf4a955_b627_4eeb_a192_ea834ed68f10.slice/crio-b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7 WatchSource:0}: Error finding container b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7: Status 404 returned error can't find the container with id b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7 Jul 13 15:15:00.314859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.314843 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 15:15:00.587470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:00.587433 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" event={"ID":"2bf4a955-b627-4eeb-a192-ea834ed68f10","Type":"ContainerStarted","Data":"b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7"} Jul 13 15:15:03.605050 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:03.605013 2554 generic.go:356] "Generic (PLEG): container finished" podID="2bf4a955-b627-4eeb-a192-ea834ed68f10" containerID="f2312a12c945f66e32346b5e03a400b33c5b0452b4835a802c696690a26e878d" exitCode=0 Jul 13 15:15:03.605437 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:03.605100 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" event={"ID":"2bf4a955-b627-4eeb-a192-ea834ed68f10","Type":"ContainerDied","Data":"f2312a12c945f66e32346b5e03a400b33c5b0452b4835a802c696690a26e878d"} Jul 13 15:15:04.754525 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:04.754500 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" Jul 13 15:15:05.615844 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:05.615775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" event={"ID":"2bf4a955-b627-4eeb-a192-ea834ed68f10","Type":"ContainerDied","Data":"b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7"} Jul 13 15:15:05.615844 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:05.615835 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732595-gqffc" Jul 13 15:15:05.615844 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:15:05.615859 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3cfacca7d184535ffe6d283f349b6d8d256cbf90826e6dab8998d07c43a2ed7" Jul 13 15:16:59.809703 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:16:59.809667 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:17:02.808053 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:17:02.808018 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:17:17.673400 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:17:17.673372 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:17:17.679471 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:17:17.679446 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:17:17.682879 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:17:17.682859 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:17:17.687974 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:17:17.687960 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:18:24.876238 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:24.876201 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:34.407374 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:34.407338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:34.505059 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:34.505028 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:34.854462 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:34.854432 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:40.919135 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:40.919080 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:40.991631 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:40.991599 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:41.166285 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:41.166258 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:41.235500 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:41.235400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:42.355583 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.355542 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:42.456447 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.456411 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:42.599350 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.599320 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:42.599969 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.599954 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf4a955-b627-4eeb-a192-ea834ed68f10" containerName="cleanup" Jul 13 15:18:42.600023 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.599972 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf4a955-b627-4eeb-a192-ea834ed68f10" containerName="cleanup" Jul 13 15:18:42.600083 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.600071 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf4a955-b627-4eeb-a192-ea834ed68f10" containerName="cleanup" Jul 13 15:18:42.603626 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.603607 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.606534 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.606478 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0b8e9a7-serving-cert\"" Jul 13 15:18:42.613423 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.613404 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:42.632946 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.632914 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:42.665450 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.665409 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.665655 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.665500 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4d5\" (UniqueName: \"kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.665655 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.665538 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.767049 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.767004 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.767231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.767100 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4d5\" (UniqueName: \"kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.767231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.767139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.767306 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:18:42.767273 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-serving-cert: secret "maas-api-e2e-ait-d0b8e9a7-serving-cert" not found Jul 13 15:18:42.767353 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:18:42.767342 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls podName:ab7b6193-42fc-4252-bfdf-38ad6f2bc678 nodeName:}" failed. No retries permitted until 2026-07-13 15:18:43.267319583 +0000 UTC m=+1586.157994668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls") pod "maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" (UID: "ab7b6193-42fc-4252-bfdf-38ad6f2bc678") : secret "maas-api-e2e-ait-d0b8e9a7-serving-cert" not found Jul 13 15:18:42.767770 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.767749 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.779208 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.779177 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4d5\" (UniqueName: \"kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:42.928599 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:42.928512 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:43.092296 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.092267 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:43.193075 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.192988 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:43.271660 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.271626 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:43.274110 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.274085 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") pod \"maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:43.517089 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.517052 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:43.868595 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:43.868569 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:43.871041 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:18:43.871013 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7b6193_42fc_4252_bfdf_38ad6f2bc678.slice/crio-9c067d57f21637d7319d97953a0405f93296f5e5d4a67277c6661c899f4a67e2 WatchSource:0}: Error finding container 9c067d57f21637d7319d97953a0405f93296f5e5d4a67277c6661c899f4a67e2: Status 404 returned error can't find the container with id 9c067d57f21637d7319d97953a0405f93296f5e5d4a67277c6661c899f4a67e2 Jul 13 15:18:44.784091 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:44.784055 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" event={"ID":"ab7b6193-42fc-4252-bfdf-38ad6f2bc678","Type":"ContainerStarted","Data":"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1"} Jul 13 15:18:44.784446 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:44.784102 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" event={"ID":"ab7b6193-42fc-4252-bfdf-38ad6f2bc678","Type":"ContainerStarted","Data":"9c067d57f21637d7319d97953a0405f93296f5e5d4a67277c6661c899f4a67e2"} Jul 13 15:18:44.784446 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:44.784134 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:44.802824 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:44.802765 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" podStartSLOduration=2.275993327 podStartE2EDuration="2.802750535s" totalImagesPullingTime="526.757208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:18:44.801455735 +0000 UTC m=+1587.692130838" watchObservedRunningTime="2026-07-13 15:18:44.802750535 +0000 UTC m=+1587.693425686" Jul 13 15:18:45.542371 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:45.542340 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:50.794434 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:50.794408 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:57.932408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:57.932373 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:57.933321 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:57.932652 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" podUID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" containerName="maas-api" containerID="cri-o://03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1" gracePeriod=30 Jul 13 15:18:57.966678 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:57.966653 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:18:58.200987 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.200961 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:58.315157 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.315118 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b4d5\" (UniqueName: \"kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5\") pod \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " Jul 13 15:18:58.315337 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.315168 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") pod \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " Jul 13 15:18:58.315337 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.315278 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca\") pod \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\" (UID: \"ab7b6193-42fc-4252-bfdf-38ad6f2bc678\") " Jul 13 15:18:58.315674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.315649 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab7b6193-42fc-4252-bfdf-38ad6f2bc678" (UID: "ab7b6193-42fc-4252-bfdf-38ad6f2bc678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:18:58.317404 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.317379 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab7b6193-42fc-4252-bfdf-38ad6f2bc678" (UID: "ab7b6193-42fc-4252-bfdf-38ad6f2bc678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:18:58.317404 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.317379 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5" (OuterVolumeSpecName: "kube-api-access-5b4d5") pod "ab7b6193-42fc-4252-bfdf-38ad6f2bc678" (UID: "ab7b6193-42fc-4252-bfdf-38ad6f2bc678"). InnerVolumeSpecName "kube-api-access-5b4d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:18:58.416245 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.416213 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-openshift-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:18:58.416245 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.416239 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4d5\" (UniqueName: \"kubernetes.io/projected/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-kube-api-access-5b4d5\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:18:58.416245 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.416249 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b6193-42fc-4252-bfdf-38ad6f2bc678-maas-api-tls\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:18:58.856273 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.856240 2554 generic.go:356] "Generic (PLEG): container finished" podID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" containerID="03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1" exitCode=0 Jul 13 15:18:58.856470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.856288 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" Jul 13 15:18:58.856470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.856326 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" event={"ID":"ab7b6193-42fc-4252-bfdf-38ad6f2bc678","Type":"ContainerDied","Data":"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1"} Jul 13 15:18:58.856470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.856380 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv" event={"ID":"ab7b6193-42fc-4252-bfdf-38ad6f2bc678","Type":"ContainerDied","Data":"9c067d57f21637d7319d97953a0405f93296f5e5d4a67277c6661c899f4a67e2"} Jul 13 15:18:58.856470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.856408 2554 scope.go:117] "RemoveContainer" containerID="03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1" Jul 13 15:18:58.866153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.866137 2554 scope.go:117] "RemoveContainer" containerID="03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1" Jul 13 15:18:58.866455 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:18:58.866427 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1\": container with ID starting with 03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1 not found: ID does not exist" containerID="03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1" Jul 13 15:18:58.866555 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.866466 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1"} err="failed to get container status \"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1\": rpc error: code = NotFound desc = could not find container \"03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1\": container with ID starting with 03931ce480645e66a03401c7c59d4766e34732cb48a7270373708f11103c32b1 not found: ID does not exist" Jul 13 15:18:58.880241 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.880219 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:58.884780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:58.884759 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0b8e9a7-fb47cf96d-twkwv"] Jul 13 15:18:59.679327 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:18:59.679289 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" path="/var/lib/kubelet/pods/ab7b6193-42fc-4252-bfdf-38ad6f2bc678/volumes" Jul 13 15:19:02.484356 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:02.484318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:08.859747 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:08.859717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:08.926932 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:08.926898 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:12.710034 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:12.709954 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:13.075524 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.075482 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:13.181008 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.180974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:13.271666 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.271630 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:19:13.272113 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.272092 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" containerName="maas-api" Jul 13 15:19:13.272226 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.272116 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" containerName="maas-api" Jul 13 15:19:13.280951 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.279977 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7b6193-42fc-4252-bfdf-38ad6f2bc678" containerName="maas-api" Jul 13 15:19:13.285402 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.285377 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.290769 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.290675 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-15019914-gw-openshift-default-dockercfg-tlgxw\"" Jul 13 15:19:13.295960 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.295938 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:19:13.328080 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.327996 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:13.350338 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350309 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350338 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350342 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350357 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gbt\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350407 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350445 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350511 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350671 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350590 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.350671 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.350633 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.451951 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.451912 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452132 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.451965 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452132 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.451985 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452132 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452002 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gbt\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452132 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452033 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452132 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452387 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452230 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452387 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452300 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452387 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452357 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452555 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452385 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452555 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452458 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452669 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452757 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452739 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.452825 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.452767 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.454403 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.454386 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.454674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.454657 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.461491 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.461468 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.461684 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.461668 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gbt\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt\") pod \"e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.597215 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.597126 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:13.741328 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.741300 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:19:13.744182 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:19:13.744151 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97e0b3d_ad6e_4a4b_b364_99959b588129.slice/crio-077492b8143547fda78e96a6f7d4fcf34b686a6a008ad024be3a646288744aca WatchSource:0}: Error finding container 077492b8143547fda78e96a6f7d4fcf34b686a6a008ad024be3a646288744aca: Status 404 returned error can't find the container with id 077492b8143547fda78e96a6f7d4fcf34b686a6a008ad024be3a646288744aca Jul 13 15:19:13.746198 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.746165 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:13.746306 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.746232 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:13.746306 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.746259 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:13.941822 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.941702 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerStarted","Data":"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349"} Jul 13 15:19:13.941822 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:13.941775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerStarted","Data":"077492b8143547fda78e96a6f7d4fcf34b686a6a008ad024be3a646288744aca"} Jul 13 15:19:14.540179 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.540141 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:14.597881 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.597846 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:14.603269 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.603244 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:14.631468 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.631426 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podStartSLOduration=1.631405207 podStartE2EDuration="1.631405207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:19:13.985335727 +0000 UTC m=+1616.876010830" watchObservedRunningTime="2026-07-13 15:19:14.631405207 +0000 UTC m=+1617.522080310" Jul 13 15:19:14.656276 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.656248 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:14.805988 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.805882 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:14.947287 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.947257 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:14.948450 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:14.948425 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:15.246543 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:15.246442 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:15.350627 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:15.350593 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:19.111267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.111230 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:19.114453 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.114438 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:19.118372 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.118348 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:19:19.124278 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.124255 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:19.206084 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.206050 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-csl47\" (UID: \"ca90f681-b4d4-4e44-a248-30011445f265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:19.300433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.300393 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:19.307232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.307206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-csl47\" (UID: \"ca90f681-b4d4-4e44-a248-30011445f265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:19.307765 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.307744 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-csl47\" (UID: \"ca90f681-b4d4-4e44-a248-30011445f265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:19.392616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.392524 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:19.426022 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.425991 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:19.771454 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.771429 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:19.773677 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:19:19.773647 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca90f681_b4d4_4e44_a248_30011445f265.slice/crio-f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11 WatchSource:0}: Error finding container f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11: Status 404 returned error can't find the container with id f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11 Jul 13 15:19:19.975566 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:19.975527 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerStarted","Data":"f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11"} Jul 13 15:19:20.982871 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:20.982833 2554 generic.go:356] "Generic (PLEG): container finished" podID="ca90f681-b4d4-4e44-a248-30011445f265" containerID="a90e83c81637ecf3fef8ba747e6394defbbeb00536d06f43e0b14a315b34d37c" exitCode=7 Jul 13 15:19:20.983321 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:20.982893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerDied","Data":"a90e83c81637ecf3fef8ba747e6394defbbeb00536d06f43e0b14a315b34d37c"} Jul 13 15:19:20.983321 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:20.983194 2554 scope.go:117] "RemoveContainer" containerID="a90e83c81637ecf3fef8ba747e6394defbbeb00536d06f43e0b14a315b34d37c" Jul 13 15:19:21.991104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:21.991021 2554 generic.go:356] "Generic (PLEG): container finished" podID="ca90f681-b4d4-4e44-a248-30011445f265" containerID="82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb" exitCode=7 Jul 13 15:19:21.991541 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:21.991110 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerDied","Data":"82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb"} Jul 13 15:19:21.991541 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:21.991178 2554 scope.go:117] "RemoveContainer" containerID="a90e83c81637ecf3fef8ba747e6394defbbeb00536d06f43e0b14a315b34d37c" Jul 13 15:19:21.991541 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:21.991331 2554 scope.go:117] "RemoveContainer" containerID="82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb" Jul 13 15:19:21.991673 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:19:21.991573 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-15019914-csl47_odh-ai-gateway-infra(ca90f681-b4d4-4e44-a248-30011445f265)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" podUID="ca90f681-b4d4-4e44-a248-30011445f265" Jul 13 15:19:22.948248 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:22.948211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:19:22.998345 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:22.998320 2554 scope.go:117] "RemoveContainer" containerID="82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb" Jul 13 15:19:22.998682 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:19:22.998527 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-15019914-csl47_odh-ai-gateway-infra(ca90f681-b4d4-4e44-a248-30011445f265)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" podUID="ca90f681-b4d4-4e44-a248-30011445f265" Jul 13 15:19:29.948619 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:29.948582 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 15:19:29.949073 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:29.948651 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 15:19:30.036102 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.036071 2554 generic.go:356] "Generic (PLEG): container finished" podID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerID="b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349" exitCode=0 Jul 13 15:19:30.036260 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.036121 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerDied","Data":"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349"} Jul 13 15:19:30.036527 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.036506 2554 scope.go:117] "RemoveContainer" containerID="b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349" Jul 13 15:19:30.037516 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.037491 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:30.037586 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.037573 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:30.037625 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:30.037610 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:19:31.043167 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:31.043137 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerStarted","Data":"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f"} Jul 13 15:19:31.597949 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:31.597914 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:31.598196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:31.598166 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 15:19:31.598315 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:31.598216 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 15:19:32.597978 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:32.597941 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 15:19:32.598358 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:32.598000 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 15:19:33.597829 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:33.597767 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 15:19:33.598036 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:33.597852 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 15:19:34.598173 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:34.598136 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 13 15:19:34.598569 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:34.598189 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 13 15:19:35.601867 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:35.601837 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:35.602306 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:35.602202 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:35.602815 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:35.602800 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:19:37.675388 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:37.675359 2554 scope.go:117] "RemoveContainer" containerID="82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb" Jul 13 15:19:38.083417 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:38.083385 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerStarted","Data":"0f2c9b8702f3c919ea2a5f3c44d274f550116bd4c3923adb7bde55d7b69bf929"} Jul 13 15:19:38.100564 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:38.100517 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" podStartSLOduration=19.10050282 podStartE2EDuration="19.10050282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:19:38.09963149 +0000 UTC m=+1640.990306592" watchObservedRunningTime="2026-07-13 15:19:38.10050282 +0000 UTC m=+1640.991177921" Jul 13 15:19:39.090125 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:39.090091 2554 generic.go:356] "Generic (PLEG): container finished" podID="ca90f681-b4d4-4e44-a248-30011445f265" containerID="0f2c9b8702f3c919ea2a5f3c44d274f550116bd4c3923adb7bde55d7b69bf929" exitCode=0 Jul 13 15:19:39.090535 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:39.090163 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerDied","Data":"0f2c9b8702f3c919ea2a5f3c44d274f550116bd4c3923adb7bde55d7b69bf929"} Jul 13 15:19:39.090535 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:39.090223 2554 scope.go:117] "RemoveContainer" containerID="82f94d7418dadf82bc98dcd53f807b987a7ce9c120dffd45a20504d6a7c280bb" Jul 13 15:19:39.120355 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:39.120328 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:40.239571 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:40.239539 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:40.420982 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:40.420899 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca\") pod \"ca90f681-b4d4-4e44-a248-30011445f265\" (UID: \"ca90f681-b4d4-4e44-a248-30011445f265\") " Jul 13 15:19:40.421276 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:40.421255 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca90f681-b4d4-4e44-a248-30011445f265" (UID: "ca90f681-b4d4-4e44-a248-30011445f265"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:19:40.522232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:40.522194 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90f681-b4d4-4e44-a248-30011445f265-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:19:41.104776 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.104748 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" Jul 13 15:19:41.104966 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.104800 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47" event={"ID":"ca90f681-b4d4-4e44-a248-30011445f265","Type":"ContainerDied","Data":"f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11"} Jul 13 15:19:41.104966 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.104845 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8efa364556fd1f803cdf31dbd3ff2d907710a6f07cf55379aa4c0eadf56dc11" Jul 13 15:19:41.146301 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.146267 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:41.149631 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.149611 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-csl47"] Jul 13 15:19:41.678854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:19:41.678823 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca90f681-b4d4-4e44-a248-30011445f265" path="/var/lib/kubelet/pods/ca90f681-b4d4-4e44-a248-30011445f265/volumes" Jul 13 15:20:02.180923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:02.180872 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:19.241292 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:19.241250 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:20:19.241763 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:19.241503 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" containerID="cri-o://87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f" gracePeriod=30 Jul 13 15:20:19.279438 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:19.279410 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:19.384305 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:19.384270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:20.604829 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:20.604781 2554 patch_prober.go:28] interesting pod/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 15:20:20.605224 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:20.604858 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 15:20:24.513651 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.513624 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:20:24.619739 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619709 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.619922 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619756 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.619922 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619804 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.619922 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619824 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.619922 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619886 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.619922 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619913 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8gbt\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.620193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619956 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.620193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.619997 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.620193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620023 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs\") pod \"e97e0b3d-ad6e-4a4b-b364-99959b588129\" (UID: \"e97e0b3d-ad6e-4a4b-b364-99959b588129\") " Jul 13 15:20:24.620193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620120 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:24.620193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620159 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:20:24.620472 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620361 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:24.620548 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620526 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:24.620740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620542 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.620878 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620864 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97e0b3d-ad6e-4a4b-b364-99959b588129-istiod-ca-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.620985 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.620971 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-credential-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.621521 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.621482 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data" (OuterVolumeSpecName: "istio-data") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:24.622213 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.622188 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:20:24.622492 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.622448 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt" (OuterVolumeSpecName: "kube-api-access-n8gbt") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "kube-api-access-n8gbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:24.622590 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.622560 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token" (OuterVolumeSpecName: "istio-token") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:20:24.622639 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.622587 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e97e0b3d-ad6e-4a4b-b364-99959b588129" (UID: "e97e0b3d-ad6e-4a4b-b364-99959b588129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:20:24.721633 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721595 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-envoy\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.721633 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721622 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-podinfo\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.721633 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721632 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.721633 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721643 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gbt\" (UniqueName: \"kubernetes.io/projected/e97e0b3d-ad6e-4a4b-b364-99959b588129-kube-api-access-n8gbt\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.721908 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721652 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-workload-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:24.721908 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:24.721660 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97e0b3d-ad6e-4a4b-b364-99959b588129-istio-data\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:20:25.343620 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.343581 2554 generic.go:356] "Generic (PLEG): container finished" podID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerID="87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f" exitCode=0 Jul 13 15:20:25.343832 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.343641 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" Jul 13 15:20:25.343832 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.343639 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerDied","Data":"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f"} Jul 13 15:20:25.343832 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.343775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp" event={"ID":"e97e0b3d-ad6e-4a4b-b364-99959b588129","Type":"ContainerDied","Data":"077492b8143547fda78e96a6f7d4fcf34b686a6a008ad024be3a646288744aca"} Jul 13 15:20:25.344017 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.343834 2554 scope.go:117] "RemoveContainer" containerID="87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f" Jul 13 15:20:25.353163 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.353147 2554 scope.go:117] "RemoveContainer" containerID="b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349" Jul 13 15:20:25.361594 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.361419 2554 scope.go:117] "RemoveContainer" containerID="87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f" Jul 13 15:20:25.361700 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:20:25.361680 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f\": container with ID starting with 87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f not found: ID does not exist" containerID="87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f" Jul 13 15:20:25.361868 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.361707 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f"} err="failed to get container status \"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f\": rpc error: code = NotFound desc = could not find container \"87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f\": container with ID starting with 87b5876a06a963ed088fc7390367e08921e7eaebdca9db40286175dccdfb185f not found: ID does not exist" Jul 13 15:20:25.361868 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.361724 2554 scope.go:117] "RemoveContainer" containerID="b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349" Jul 13 15:20:25.361999 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:20:25.361961 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349\": container with ID starting with b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349 not found: ID does not exist" containerID="b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349" Jul 13 15:20:25.361999 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.361978 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349"} err="failed to get container status \"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349\": rpc error: code = NotFound desc = could not find container \"b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349\": container with ID starting with b6b16b71f812859be09cd458ca76536568cde28736898d7246e45069ca66a349 not found: ID does not exist" Jul 13 15:20:25.369180 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.369160 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:20:25.374321 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.374299 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-15019914-gw-openshift-default-68666bc98c-jsdbp"] Jul 13 15:20:25.422655 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.422620 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:25.525244 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.525211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:25.662219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.662142 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:25.678945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:25.678918 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" path="/var/lib/kubelet/pods/e97e0b3d-ad6e-4a4b-b364-99959b588129/volumes" Jul 13 15:20:31.665921 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:31.665879 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:31.764616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:31.764579 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:31.919035 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:31.918942 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:32.024594 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:32.024555 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:34.520983 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:34.520946 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:34.622095 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:34.622062 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:20:35.212265 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:20:35.212229 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:38.870177 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:38.870144 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:43.544291 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:43.544254 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:49.868856 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:49.868821 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:50.005098 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:50.005067 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:54.685589 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.685552 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:54.984779 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.984752 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:21:54.985231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985217 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985234 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985242 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985248 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985262 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985268 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985277 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985282 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985282 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985354 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985362 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:21:54.985486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985367 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:21:54.985486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.985376 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:21:54.988671 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.988651 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:54.991954 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:54.991930 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f99515a5-openshift-default-dockercfg-5ggw4\"" Jul 13 15:21:55.004184 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.004160 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:21:55.042260 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.042237 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:55.110715 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.110715 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110714 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.110939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.110939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110820 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.110939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110844 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rfh\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.110939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.111088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.110991 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.111088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.111020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.111088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.111051 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.211846 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211816 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.211846 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211880 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.211979 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212297 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212117 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212297 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212161 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rfh\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212297 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212260 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212453 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212292 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212453 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212356 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212624 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212582 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.212738 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.213009 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.212992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.214840 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.214819 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.214964 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.214948 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.222544 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.222523 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rfh\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.222645 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.222548 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token\") pod \"e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.299854 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.299818 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:55.449444 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.449388 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:21:55.451645 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:21:55.451617 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82da2f87_f85c_4c38_9728_dfefbac8a698.slice/crio-c75bcd642909c65634027a75af6724979a59c5ee47c70efcd632fface5d253b3 WatchSource:0}: Error finding container c75bcd642909c65634027a75af6724979a59c5ee47c70efcd632fface5d253b3: Status 404 returned error can't find the container with id c75bcd642909c65634027a75af6724979a59c5ee47c70efcd632fface5d253b3 Jul 13 15:21:55.453818 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.453761 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:21:55.453923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.453866 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:21:55.453923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.453913 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:21:55.812444 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.812400 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerStarted","Data":"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e"} Jul 13 15:21:55.812837 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.812468 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerStarted","Data":"c75bcd642909c65634027a75af6724979a59c5ee47c70efcd632fface5d253b3"} Jul 13 15:21:55.841150 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:55.841096 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podStartSLOduration=1.8410805780000001 podStartE2EDuration="1.841080578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:21:55.838509338 +0000 UTC m=+1778.729184467" watchObservedRunningTime="2026-07-13 15:21:55.841080578 +0000 UTC m=+1778.731755657" Jul 13 15:21:56.241097 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.240926 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:56.300020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.299984 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:56.305129 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.305093 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:56.340249 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.340204 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:56.530383 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.530349 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:56.820279 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.820180 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:56.821866 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.821769 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:21:56.988966 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:56.988934 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:57.134855 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:57.134752 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:21:57.215118 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:21:57.215082 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:00.577603 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.577563 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:00.578091 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.578054 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:22:00.578091 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.578067 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97e0b3d-ad6e-4a4b-b364-99959b588129" containerName="istio-proxy" Jul 13 15:22:00.578212 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.578143 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca90f681-b4d4-4e44-a248-30011445f265" containerName="revoke-keys" Jul 13 15:22:00.581643 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.581621 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:00.584309 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.584286 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:22:00.589968 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.589946 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:00.667859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.667826 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp\" (UID: \"c3c5c9aa-4150-4d86-9e8e-634214d32918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:00.769239 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.769202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp\" (UID: \"c3c5c9aa-4150-4d86-9e8e-634214d32918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:00.769808 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.769765 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp\" (UID: \"c3c5c9aa-4150-4d86-9e8e-634214d32918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:00.893719 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:00.893639 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:01.085293 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.085259 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:01.086754 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:22:01.086728 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c5c9aa_4150_4d86_9e8e_634214d32918.slice/crio-435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca WatchSource:0}: Error finding container 435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca: Status 404 returned error can't find the container with id 435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca Jul 13 15:22:01.583847 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.583772 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:01.678113 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.678068 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:01.857924 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.857827 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerStarted","Data":"45bc4ec1e9913f5324bce878dc64720f682ca978edd1ea4b6d76d03c646539ea"} Jul 13 15:22:01.857924 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.857885 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerStarted","Data":"435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca"} Jul 13 15:22:01.881727 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:01.881679 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" podStartSLOduration=1.8816642479999999 podStartE2EDuration="1.881664248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:22:01.879564598 +0000 UTC m=+1784.770239700" watchObservedRunningTime="2026-07-13 15:22:01.881664248 +0000 UTC m=+1784.772339350" Jul 13 15:22:02.864192 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:02.864162 2554 generic.go:356] "Generic (PLEG): container finished" podID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerID="45bc4ec1e9913f5324bce878dc64720f682ca978edd1ea4b6d76d03c646539ea" exitCode=7 Jul 13 15:22:02.864596 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:02.864258 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerDied","Data":"45bc4ec1e9913f5324bce878dc64720f682ca978edd1ea4b6d76d03c646539ea"} Jul 13 15:22:02.864596 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:02.864444 2554 scope.go:117] "RemoveContainer" containerID="45bc4ec1e9913f5324bce878dc64720f682ca978edd1ea4b6d76d03c646539ea" Jul 13 15:22:03.872215 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:03.872174 2554 generic.go:356] "Generic (PLEG): container finished" podID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerID="2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3" exitCode=7 Jul 13 15:22:03.872585 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:03.872260 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerDied","Data":"2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3"} Jul 13 15:22:03.872585 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:03.872320 2554 scope.go:117] "RemoveContainer" containerID="45bc4ec1e9913f5324bce878dc64720f682ca978edd1ea4b6d76d03c646539ea" Jul 13 15:22:03.872585 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:03.872557 2554 scope.go:117] "RemoveContainer" containerID="2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3" Jul 13 15:22:03.872859 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:22:03.872840 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp_odh-ai-gateway-infra(c3c5c9aa-4150-4d86-9e8e-634214d32918)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" Jul 13 15:22:04.505481 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:04.505444 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:04.880001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:04.879973 2554 scope.go:117] "RemoveContainer" containerID="2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3" Jul 13 15:22:04.880344 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:22:04.880232 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp_odh-ai-gateway-infra(c3c5c9aa-4150-4d86-9e8e-634214d32918)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" Jul 13 15:22:06.582329 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:06.582289 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:07.899688 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.899656 2554 generic.go:356] "Generic (PLEG): container finished" podID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerID="9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e" exitCode=0 Jul 13 15:22:07.900117 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.899736 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerDied","Data":"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e"} Jul 13 15:22:07.900187 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.900172 2554 scope.go:117] "RemoveContainer" containerID="9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e" Jul 13 15:22:07.901182 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.901152 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:22:07.901290 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.901238 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:22:07.901290 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:07.901285 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:22:08.907940 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:08.907907 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerStarted","Data":"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac"} Jul 13 15:22:09.300896 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:09.300854 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:22:09.302255 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:09.302227 2554 patch_prober.go:28] interesting pod/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 15:22:09.302394 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:09.302276 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 15:22:10.301160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:10.301079 2554 patch_prober.go:28] interesting pod/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 15:22:10.301160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:10.301132 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 15:22:11.300780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:11.300743 2554 patch_prober.go:28] interesting pod/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 15:22:11.300975 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:11.300826 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 15:22:11.820730 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:11.820690 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:22:12.300801 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:12.300756 2554 patch_prober.go:28] interesting pod/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 15:22:12.300971 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:12.300829 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 15:22:13.304506 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:13.304473 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:22:13.305466 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:13.305447 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:22:15.673485 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:15.673449 2554 scope.go:117] "RemoveContainer" containerID="2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3" Jul 13 15:22:15.951169 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:15.951140 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerStarted","Data":"159ca6722d0198fc5204abfa76866439171b1d212245714e92870ecc28711922"} Jul 13 15:22:16.958372 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:16.958338 2554 generic.go:356] "Generic (PLEG): container finished" podID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerID="159ca6722d0198fc5204abfa76866439171b1d212245714e92870ecc28711922" exitCode=0 Jul 13 15:22:16.958858 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:16.958419 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerDied","Data":"159ca6722d0198fc5204abfa76866439171b1d212245714e92870ecc28711922"} Jul 13 15:22:16.958858 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:16.958493 2554 scope.go:117] "RemoveContainer" containerID="2b15412cdfb2670761fd8e06f9a4b7fe8a2d4050ed99e6cd693b8f7f4adcbec3" Jul 13 15:22:16.989886 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:16.989854 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:17.705933 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:17.705905 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:22:17.711401 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:17.711378 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:22:17.716278 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:17.716258 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:22:17.721481 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:17.721464 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:22:18.105020 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.104992 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:18.245047 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.244952 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca\") pod \"c3c5c9aa-4150-4d86-9e8e-634214d32918\" (UID: \"c3c5c9aa-4150-4d86-9e8e-634214d32918\") " Jul 13 15:22:18.245388 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.245353 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3c5c9aa-4150-4d86-9e8e-634214d32918" (UID: "c3c5c9aa-4150-4d86-9e8e-634214d32918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:22:18.346513 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.346477 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c5c9aa-4150-4d86-9e8e-634214d32918-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:22:18.973613 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.973582 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" Jul 13 15:22:18.973779 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.973581 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp" event={"ID":"c3c5c9aa-4150-4d86-9e8e-634214d32918","Type":"ContainerDied","Data":"435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca"} Jul 13 15:22:18.973779 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:18.973719 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="435f2b47eb947f5c1715a884953a874face8a6d898c32b9894e812b0154a18ca" Jul 13 15:22:19.010603 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:19.010572 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:19.029734 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:19.029702 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-gvxcp"] Jul 13 15:22:19.103102 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:19.103064 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:19.200516 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:19.200484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:22:19.679660 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:19.679632 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" path="/var/lib/kubelet/pods/c3c5c9aa-4150-4d86-9e8e-634214d32918/volumes" Jul 13 15:22:50.643698 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.643655 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb"] Jul 13 15:22:50.644172 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644157 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.644219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644175 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.644219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644192 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.644219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644197 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.644324 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644264 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.644324 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.644273 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:22:50.646359 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.646344 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:50.648934 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.648913 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:22:50.655340 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.655309 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb"] Jul 13 15:22:50.740342 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.740303 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb\" (UID: \"bba4d722-9097-4727-8cd7-01f2409b5f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:50.841595 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.841557 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb\" (UID: \"bba4d722-9097-4727-8cd7-01f2409b5f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:50.842223 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.842206 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb\" (UID: \"bba4d722-9097-4727-8cd7-01f2409b5f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:50.959863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:50.959765 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:51.308487 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:51.308462 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb"] Jul 13 15:22:51.309919 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:22:51.309895 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba4d722_9097_4727_8cd7_01f2409b5f5c.slice/crio-5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3 WatchSource:0}: Error finding container 5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3: Status 404 returned error can't find the container with id 5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3 Jul 13 15:22:52.153095 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:52.153062 2554 generic.go:356] "Generic (PLEG): container finished" podID="bba4d722-9097-4727-8cd7-01f2409b5f5c" containerID="b84efcfa8beaf8b27348a0055468cb744d21567b2982655be5ea52b02683385b" exitCode=0 Jul 13 15:22:52.153437 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:52.153148 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" event={"ID":"bba4d722-9097-4727-8cd7-01f2409b5f5c","Type":"ContainerDied","Data":"b84efcfa8beaf8b27348a0055468cb744d21567b2982655be5ea52b02683385b"} Jul 13 15:22:52.153437 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:52.153201 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" event={"ID":"bba4d722-9097-4727-8cd7-01f2409b5f5c","Type":"ContainerStarted","Data":"5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3"} Jul 13 15:22:53.301268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:53.301242 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:53.463998 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:53.463920 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca\") pod \"bba4d722-9097-4727-8cd7-01f2409b5f5c\" (UID: \"bba4d722-9097-4727-8cd7-01f2409b5f5c\") " Jul 13 15:22:53.464264 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:53.464243 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bba4d722-9097-4727-8cd7-01f2409b5f5c" (UID: "bba4d722-9097-4727-8cd7-01f2409b5f5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:22:53.565480 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:53.565449 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba4d722-9097-4727-8cd7-01f2409b5f5c-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:22:54.165540 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:54.165507 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" event={"ID":"bba4d722-9097-4727-8cd7-01f2409b5f5c","Type":"ContainerDied","Data":"5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3"} Jul 13 15:22:54.165540 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:54.165526 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb" Jul 13 15:22:54.165755 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:22:54.165555 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea9fd4c8522a378926c48032b2fb95bd8938b0375a8973032da64f2aa670bd3" Jul 13 15:23:01.201839 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:01.201780 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:05.738438 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:05.738404 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:11.673224 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:11.673186 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:11.738264 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:11.738230 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:15.928311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:15.928274 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:23:15.928701 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:15.928482 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" containerID="cri-o://57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac" gracePeriod=30 Jul 13 15:23:15.948245 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:15.948212 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:16.605690 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:16.605658 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:16.793233 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:16.793200 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:16.921560 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:16.921470 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:18.384564 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:18.384528 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:21.183655 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.183633 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:23:21.212950 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.212863 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.212950 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.212915 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.212955 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.212991 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213043 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213086 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24rfh\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213160 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213145 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:23:21.213421 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213179 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213421 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213223 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213421 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213246 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs\") pod \"82da2f87-f85c-4c38-9728-dfefbac8a698\" (UID: \"82da2f87-f85c-4c38-9728-dfefbac8a698\") " Jul 13 15:23:21.213576 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213500 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-credential-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.213745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213721 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:23:21.213745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213742 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:23:21.214009 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.213950 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:23:21.214888 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.214855 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data" (OuterVolumeSpecName: "istio-data") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:23:21.215649 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.215614 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:23:21.216184 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.216159 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:23:21.216184 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.216167 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh" (OuterVolumeSpecName: "kube-api-access-24rfh") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "kube-api-access-24rfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:23:21.216308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.216202 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token" (OuterVolumeSpecName: "istio-token") pod "82da2f87-f85c-4c38-9728-dfefbac8a698" (UID: "82da2f87-f85c-4c38-9728-dfefbac8a698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:23:21.313835 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.313773 2554 generic.go:356] "Generic (PLEG): container finished" podID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerID="57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac" exitCode=0 Jul 13 15:23:21.314030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.313862 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" Jul 13 15:23:21.314030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.313870 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerDied","Data":"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac"} Jul 13 15:23:21.314030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.313916 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6" event={"ID":"82da2f87-f85c-4c38-9728-dfefbac8a698","Type":"ContainerDied","Data":"c75bcd642909c65634027a75af6724979a59c5ee47c70efcd632fface5d253b3"} Jul 13 15:23:21.314030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.313936 2554 scope.go:117] "RemoveContainer" containerID="57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314115 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-envoy\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314134 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-podinfo\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314147 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314162 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314175 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rfh\" (UniqueName: \"kubernetes.io/projected/82da2f87-f85c-4c38-9728-dfefbac8a698-kube-api-access-24rfh\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314189 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82da2f87-f85c-4c38-9728-dfefbac8a698-istiod-ca-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314200 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-istio-data\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.314217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.314212 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82da2f87-f85c-4c38-9728-dfefbac8a698-workload-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:23:21.324736 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.324714 2554 scope.go:117] "RemoveContainer" containerID="9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e" Jul 13 15:23:21.333523 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.333506 2554 scope.go:117] "RemoveContainer" containerID="57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac" Jul 13 15:23:21.333764 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:23:21.333746 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac\": container with ID starting with 57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac not found: ID does not exist" containerID="57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac" Jul 13 15:23:21.333875 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.333774 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac"} err="failed to get container status \"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac\": rpc error: code = NotFound desc = could not find container \"57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac\": container with ID starting with 57c9ed243997942665b9b14fb900ee8403cf8738f94a83a3ccb89e213b7aadac not found: ID does not exist" Jul 13 15:23:21.333875 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.333808 2554 scope.go:117] "RemoveContainer" containerID="9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e" Jul 13 15:23:21.334052 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:23:21.334034 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e\": container with ID starting with 9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e not found: ID does not exist" containerID="9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e" Jul 13 15:23:21.334106 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.334062 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e"} err="failed to get container status \"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e\": rpc error: code = NotFound desc = could not find container \"9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e\": container with ID starting with 9844ef8739638a1c01d832bc656ad7d4acb7e4644279a084389d90a3fbf8a72e not found: ID does not exist" Jul 13 15:23:21.338813 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.338771 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:23:21.344563 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.344542 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f99515a5-openshift-default-755cfb59f9-vkkr6"] Jul 13 15:23:21.679923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:21.679894 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" path="/var/lib/kubelet/pods/82da2f87-f85c-4c38-9728-dfefbac8a698/volumes" Jul 13 15:23:22.360017 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:22.359981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:24.796031 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:24.795984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:24.904327 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:24.904290 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:25.053467 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:25.053388 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:25.193344 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:25.193300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:25.565620 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:25.565587 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:51.550509 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:51.550479 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:57.537094 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:57.537058 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:57.657554 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:57.657526 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:23:57.901178 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:23:57.901090 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:13.692363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:13.692329 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:19.692113 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:19.692079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:19.794435 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:19.794402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:19.892708 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:19.892677 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:30.467829 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:30.467767 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:31.295296 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:31.295263 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:37.430923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:37.430890 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:37.534060 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:37.534026 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:37.626895 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:37.626865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:37.726831 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:37.726713 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:38.541334 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:38.541299 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:38.650981 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:38.650946 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:38.764367 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:38.764336 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:40.451126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:40.451097 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:46.534985 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:46.534943 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:46.611158 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:46.611124 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:47.520343 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:47.520293 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:47.620593 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:47.620552 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:53.398129 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:53.398096 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:53.497917 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:53.497886 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:53.785064 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:53.785025 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:53.982940 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:53.982909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:54.725479 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:54.725439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:57.477231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:57.477193 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:57.577443 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:57.577412 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:57.655617 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:57.655559 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:58.245177 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:58.245136 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:24:58.349681 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:24:58.349652 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:02.063461 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:02.063426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:02.710176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:02.710140 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:02.817154 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:02.817117 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:03.188956 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:03.188920 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:09.392519 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:09.392487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:09.471943 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:09.471910 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:20.342240 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:20.342151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:24.919769 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:24.919733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:31.013893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:31.013852 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:31.115304 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:31.115274 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:35.600521 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:35.600490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:35.847501 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:35.847457 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:35.958299 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:35.958204 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:36.061077 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:36.061047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:36.186966 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:36.186932 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:42.478889 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:42.478854 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:42.584931 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:42.584896 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:42.680953 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:42.680921 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:42.882476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:42.882442 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:48.696005 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:48.695974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:48.788376 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:48.788346 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:55.656297 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:55.656244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:55.984054 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:55.983971 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:56.658057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:56.658023 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:58.822219 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:58.822183 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:58.917364 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:58.917331 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:58.994483 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:58.994454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:59.103711 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:59.103622 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:59.649665 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:59.649631 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:25:59.747753 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:25:59.747720 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:26:03.436626 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:03.436589 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:26:17.861675 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:17.861645 2554 scope.go:117] "RemoveContainer" containerID="0f2c9b8702f3c919ea2a5f3c44d274f550116bd4c3923adb7bde55d7b69bf929" Jul 13 15:26:24.522488 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:24.522451 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:26:52.077129 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077043 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc"] Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077467 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077478 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077489 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077494 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077509 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba4d722-9097-4727-8cd7-01f2409b5f5c" containerName="revoke-keys" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077515 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba4d722-9097-4727-8cd7-01f2409b5f5c" containerName="revoke-keys" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077527 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.077580 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077533 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.077840 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077611 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba4d722-9097-4727-8cd7-01f2409b5f5c" containerName="revoke-keys" Jul 13 15:26:52.077840 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077619 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c5c9aa-4150-4d86-9e8e-634214d32918" containerName="revoke-keys" Jul 13 15:26:52.077840 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077627 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.077840 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.077632 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="82da2f87-f85c-4c38-9728-dfefbac8a698" containerName="istio-proxy" Jul 13 15:26:52.079644 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.079621 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:52.082416 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.082393 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:26:52.090241 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.090218 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc"] Jul 13 15:26:52.142183 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.142156 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc\" (UID: \"cc41157c-5c08-464f-90fe-aed10e28f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:52.243070 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.243038 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc\" (UID: \"cc41157c-5c08-464f-90fe-aed10e28f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:52.243894 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.243860 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc\" (UID: \"cc41157c-5c08-464f-90fe-aed10e28f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:52.391243 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.391157 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:52.541494 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:52.541472 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc"] Jul 13 15:26:52.543798 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:26:52.543753 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc41157c_5c08_464f_90fe_aed10e28f279.slice/crio-14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef WatchSource:0}: Error finding container 14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef: Status 404 returned error can't find the container with id 14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef Jul 13 15:26:53.444381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:53.444348 2554 generic.go:356] "Generic (PLEG): container finished" podID="cc41157c-5c08-464f-90fe-aed10e28f279" containerID="9fde797243962800c26f6ac3ff5886d98b0aee5ceb4a2b39d1dcff1be1d68b4f" exitCode=0 Jul 13 15:26:53.444806 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:53.444437 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" event={"ID":"cc41157c-5c08-464f-90fe-aed10e28f279","Type":"ContainerDied","Data":"9fde797243962800c26f6ac3ff5886d98b0aee5ceb4a2b39d1dcff1be1d68b4f"} Jul 13 15:26:53.444806 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:53.444486 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" event={"ID":"cc41157c-5c08-464f-90fe-aed10e28f279","Type":"ContainerStarted","Data":"14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef"} Jul 13 15:26:54.597231 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:54.597207 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:54.664638 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:54.664600 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca\") pod \"cc41157c-5c08-464f-90fe-aed10e28f279\" (UID: \"cc41157c-5c08-464f-90fe-aed10e28f279\") " Jul 13 15:26:54.664978 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:54.664955 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc41157c-5c08-464f-90fe-aed10e28f279" (UID: "cc41157c-5c08-464f-90fe-aed10e28f279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:26:54.766099 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:54.766071 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41157c-5c08-464f-90fe-aed10e28f279-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:26:55.461735 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:55.461709 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" Jul 13 15:26:55.461918 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:55.461760 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc" event={"ID":"cc41157c-5c08-464f-90fe-aed10e28f279","Type":"ContainerDied","Data":"14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef"} Jul 13 15:26:55.461918 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:55.461837 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f4a68b112ae7ec9ec7291ad0d4ec66fbfb87c0e3aed24b428c6d3b2f6c59ef" Jul 13 15:26:58.677563 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:26:58.677530 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:02.629886 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:02.629850 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:05.289322 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:05.289292 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:05.389696 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:05.389660 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:07.170839 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:07.170780 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:12.255251 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:12.255215 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:17.738674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:17.738640 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:27:17.743845 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:17.743823 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:27:17.750574 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:17.750555 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:27:17.755750 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:17.755732 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:27:48.874970 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:48.874933 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:49.981554 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:49.981520 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:55.587210 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:55.587172 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:56.196636 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:56.196595 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:56.296487 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:56.296447 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:59.501908 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:59.501874 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:27:59.721048 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:27:59.721020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:00.802090 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:00.802054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:00.902139 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:00.902105 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:01.006410 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:01.006382 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:01.815040 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:01.815005 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:01.911761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:01.911725 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:03.459173 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:03.459142 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:09.596233 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:09.596202 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:09.695720 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:09.695690 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:10.287068 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:10.286986 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:16.171208 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:16.171175 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:16.292495 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:16.292462 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:16.407714 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:16.407684 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:17.888465 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:17.888429 2554 scope.go:117] "RemoveContainer" containerID="159ca6722d0198fc5204abfa76866439171b1d212245714e92870ecc28711922" Jul 13 15:28:27.015448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:27.015412 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:27.382140 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:27.382062 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:48.357235 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:48.357202 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:54.312364 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:54.312331 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:54.415301 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:54.415272 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:54.721399 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:54.721323 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:55.034530 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:55.034492 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:55.322065 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:55.321975 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:57.695476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:57.695442 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:57.795845 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:57.795806 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:57.963041 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:57.962945 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:58.172333 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:58.172300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:58.474025 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:58.473992 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:28:58.572933 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:28:58.572900 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:02.223239 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.223202 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:02.566124 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.566091 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:02.846850 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.846752 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:29:02.847466 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.847448 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc41157c-5c08-464f-90fe-aed10e28f279" containerName="revoke-keys" Jul 13 15:29:02.847547 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.847468 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc41157c-5c08-464f-90fe-aed10e28f279" containerName="revoke-keys" Jul 13 15:29:02.847588 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.847565 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc41157c-5c08-464f-90fe-aed10e28f279" containerName="revoke-keys" Jul 13 15:29:02.851352 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.851328 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.855425 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.855401 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f68651-openshift-default-dockercfg-c9khh\"" Jul 13 15:29:02.867307 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.867282 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:29:02.872913 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.872876 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:02.933539 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933504 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933550 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxx5\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933602 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933990 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933744 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.933990 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.933913 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:02.934102 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:02.934025 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034549 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034517 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034681 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034707 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxx5\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034761 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034733 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034776 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.034957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.034936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035170 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035000 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035312 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035288 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035345 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035366 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035535 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.035623 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.035563 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.037120 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.037084 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.037698 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.037678 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.044005 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.043985 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxx5\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.044079 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.043989 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token\") pod \"e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.103250 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.103156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:03.162477 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.162444 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:03.318135 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.318107 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:29:03.320573 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:29:03.320541 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9182f9b3_7a6f_4f20_913e_e9ed2d0e0a8d.slice/crio-fe10c3d4c7ec9c140ecc1fed3693ae16d9694daf360c5c31c8e3c32ca7c2ed33 WatchSource:0}: Error finding container fe10c3d4c7ec9c140ecc1fed3693ae16d9694daf360c5c31c8e3c32ca7c2ed33: Status 404 returned error can't find the container with id fe10c3d4c7ec9c140ecc1fed3693ae16d9694daf360c5c31c8e3c32ca7c2ed33 Jul 13 15:29:03.323009 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.322972 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:03.323086 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.323062 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:03.323128 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:03.323103 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:04.141425 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:04.141389 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerStarted","Data":"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36"} Jul 13 15:29:04.141605 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:04.141455 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerStarted","Data":"fe10c3d4c7ec9c140ecc1fed3693ae16d9694daf360c5c31c8e3c32ca7c2ed33"} Jul 13 15:29:04.162861 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:04.162831 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:04.163676 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:04.163630 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podStartSLOduration=2.163615724 podStartE2EDuration="2.163615724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:29:04.161970166 +0000 UTC m=+2207.052645269" watchObservedRunningTime="2026-07-13 15:29:04.163615724 +0000 UTC m=+2207.054290884" Jul 13 15:29:04.167522 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:04.167500 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:05.146420 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:05.146392 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:05.147382 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:05.147357 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:06.101673 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.101634 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:06.178849 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.178805 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:06.246351 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.246311 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:06.346980 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.346952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:06.736562 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.736532 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:06.840994 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:06.840963 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:10.601375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:10.601341 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:15.048505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:15.048471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:15.148742 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:15.148695 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:17.218729 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:17.218677 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:20.148644 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.148566 2554 patch_prober.go:28] interesting pod/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 13 15:29:20.149043 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.148635 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 13 15:29:20.226762 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.226679 2554 generic.go:356] "Generic (PLEG): container finished" podID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerID="c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36" exitCode=0 Jul 13 15:29:20.226959 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.226752 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerDied","Data":"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36"} Jul 13 15:29:20.227312 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.227283 2554 scope.go:117] "RemoveContainer" containerID="c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36" Jul 13 15:29:20.228420 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.228390 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:20.228533 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.228480 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:20.228598 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.228556 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:29:20.345053 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.345022 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:20.446268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:20.446209 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:21.238907 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:21.238865 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerStarted","Data":"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188"} Jul 13 15:29:22.163217 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:22.163179 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:22.163443 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:22.163421 2554 patch_prober.go:28] interesting pod/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 13 15:29:22.163506 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:22.163488 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 13 15:29:23.163155 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:23.163114 2554 patch_prober.go:28] interesting pod/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 13 15:29:23.163629 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:23.163179 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 13 15:29:24.163125 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:24.163087 2554 patch_prober.go:28] interesting pod/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 13 15:29:24.163323 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:24.163148 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 13 15:29:25.168210 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:25.168183 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:25.168665 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:25.168619 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:25.169420 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:25.169400 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:29:28.486122 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:28.486085 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:28.591490 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:28.591453 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:29:42.802450 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:29:42.802360 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:00.134801 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.134753 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq"] Jul 13 15:30:00.138758 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.138736 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc"] Jul 13 15:30:00.138902 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.138881 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" Jul 13 15:30:00.141468 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.141436 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:30:00.142042 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.142024 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" Jul 13 15:30:00.142141 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.142100 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst"] Jul 13 15:30:00.146025 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.146004 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" Jul 13 15:30:00.146974 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.146935 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq"] Jul 13 15:30:00.149897 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.149876 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst"] Jul 13 15:30:00.153136 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.153115 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc"] Jul 13 15:30:00.154053 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.154031 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" Jul 13 15:30:00.164911 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.164890 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" Jul 13 15:30:00.213538 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.213514 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" Jul 13 15:30:00.404240 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.404220 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst"] Jul 13 15:30:00.405640 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:30:00.405612 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3030602b_edef_4f56_9f73_d50fd834817d.slice/crio-750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635 WatchSource:0}: Error finding container 750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635: Status 404 returned error can't find the container with id 750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635 Jul 13 15:30:00.455859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.455811 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" event={"ID":"3030602b-edef-4f56-9f73-d50fd834817d","Type":"ContainerStarted","Data":"750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635"} Jul 13 15:30:00.534503 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.534479 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq"] Jul 13 15:30:00.537988 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:30:00.537963 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcd203e_3fa9_42b2_932b_74966a1eba49.slice/crio-a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53 WatchSource:0}: Error finding container a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53: Status 404 returned error can't find the container with id a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53 Jul 13 15:30:00.581411 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.581389 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc"] Jul 13 15:30:00.583140 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:30:00.583120 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fae1dd_a9db_465e_9250_759c40c0d175.slice/crio-75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0 WatchSource:0}: Error finding container 75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0: Status 404 returned error can't find the container with id 75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0 Jul 13 15:30:00.707515 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:00.707427 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:01.465076 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.464971 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" event={"ID":"3030602b-edef-4f56-9f73-d50fd834817d","Type":"ContainerStarted","Data":"2c69e09556c5bf3f16dbb0f1d2211c9bf935098c733370865b99dcb5874d3ddb"} Jul 13 15:30:01.466728 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.466705 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" event={"ID":"b2fae1dd-a9db-465e-9250-759c40c0d175","Type":"ContainerStarted","Data":"e809ed1ea6d78f37d8ff6845799330661aa14d6a46e72959ed891f64a4d834a4"} Jul 13 15:30:01.466879 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.466753 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" event={"ID":"b2fae1dd-a9db-465e-9250-759c40c0d175","Type":"ContainerStarted","Data":"75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0"} Jul 13 15:30:01.468463 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.468438 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" event={"ID":"bbcd203e-3fa9-42b2-932b-74966a1eba49","Type":"ContainerStarted","Data":"268f504d83e9ded7baecb33dc36157bd9ea18036df023da43b044c9c9a90a8aa"} Jul 13 15:30:01.468575 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.468483 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" event={"ID":"bbcd203e-3fa9-42b2-932b-74966a1eba49","Type":"ContainerStarted","Data":"a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53"} Jul 13 15:30:01.485423 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.485379 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" podStartSLOduration=1.485365546 podStartE2EDuration="1.485365546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:30:01.483349382 +0000 UTC m=+2264.374024494" watchObservedRunningTime="2026-07-13 15:30:01.485365546 +0000 UTC m=+2264.376040648" Jul 13 15:30:01.501682 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.501641 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" podStartSLOduration=1.501629651 podStartE2EDuration="1.501629651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:30:01.500808312 +0000 UTC m=+2264.391483407" watchObservedRunningTime="2026-07-13 15:30:01.501629651 +0000 UTC m=+2264.392304753" Jul 13 15:30:01.516036 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:01.515996 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" podStartSLOduration=1.515982516 podStartE2EDuration="1.515982516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:30:01.51582413 +0000 UTC m=+2264.406499233" watchObservedRunningTime="2026-07-13 15:30:01.515982516 +0000 UTC m=+2264.406657617" Jul 13 15:30:02.045484 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.045449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:02.134741 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.134682 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw"] Jul 13 15:30:02.139110 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.139084 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:02.146708 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.146678 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw"] Jul 13 15:30:02.197068 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.197034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw\" (UID: \"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:02.298556 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.298455 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw\" (UID: \"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:02.299233 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.299209 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw\" (UID: \"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:02.453841 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.453777 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:02.476965 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.476931 2554 generic.go:356] "Generic (PLEG): container finished" podID="3030602b-edef-4f56-9f73-d50fd834817d" containerID="2c69e09556c5bf3f16dbb0f1d2211c9bf935098c733370865b99dcb5874d3ddb" exitCode=0 Jul 13 15:30:02.477404 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.477031 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" event={"ID":"3030602b-edef-4f56-9f73-d50fd834817d","Type":"ContainerDied","Data":"2c69e09556c5bf3f16dbb0f1d2211c9bf935098c733370865b99dcb5874d3ddb"} Jul 13 15:30:02.478879 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.478819 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2fae1dd-a9db-465e-9250-759c40c0d175" containerID="e809ed1ea6d78f37d8ff6845799330661aa14d6a46e72959ed891f64a4d834a4" exitCode=0 Jul 13 15:30:02.478995 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.478922 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" event={"ID":"b2fae1dd-a9db-465e-9250-759c40c0d175","Type":"ContainerDied","Data":"e809ed1ea6d78f37d8ff6845799330661aa14d6a46e72959ed891f64a4d834a4"} Jul 13 15:30:02.480654 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.480632 2554 generic.go:356] "Generic (PLEG): container finished" podID="bbcd203e-3fa9-42b2-932b-74966a1eba49" containerID="268f504d83e9ded7baecb33dc36157bd9ea18036df023da43b044c9c9a90a8aa" exitCode=0 Jul 13 15:30:02.480762 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.480725 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" event={"ID":"bbcd203e-3fa9-42b2-932b-74966a1eba49","Type":"ContainerDied","Data":"268f504d83e9ded7baecb33dc36157bd9ea18036df023da43b044c9c9a90a8aa"} Jul 13 15:30:02.607441 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.607415 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw"] Jul 13 15:30:02.609276 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:30:02.609245 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3695726_9bd0_4aa1_a08b_cf0e3fb036d7.slice/crio-27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7 WatchSource:0}: Error finding container 27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7: Status 404 returned error can't find the container with id 27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7 Jul 13 15:30:02.798653 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.798612 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:02.899642 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:02.899557 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:03.488499 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.488465 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" event={"ID":"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7","Type":"ContainerStarted","Data":"f76852a0364f72db73c7f3265e39b4aea4c45fae0ec174c2302c2d479b79829d"} Jul 13 15:30:03.489004 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.488518 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" event={"ID":"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7","Type":"ContainerStarted","Data":"27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7"} Jul 13 15:30:03.507916 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.507849 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" podStartSLOduration=1.507821858 podStartE2EDuration="1.507821858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:30:03.504118816 +0000 UTC m=+2266.394793919" watchObservedRunningTime="2026-07-13 15:30:03.507821858 +0000 UTC m=+2266.398496960" Jul 13 15:30:03.727093 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.727067 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" Jul 13 15:30:03.730033 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.730010 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" Jul 13 15:30:03.734779 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:03.734758 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" Jul 13 15:30:04.494985 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.494959 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" Jul 13 15:30:04.495433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.494958 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc" event={"ID":"b2fae1dd-a9db-465e-9250-759c40c0d175","Type":"ContainerDied","Data":"75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0"} Jul 13 15:30:04.495433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.495075 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c69cfc5862d08f738113650fb8a05dd1c74a17df62e798eb235c32604880e0" Jul 13 15:30:04.496329 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.496314 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" Jul 13 15:30:04.496448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.496364 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq" event={"ID":"bbcd203e-3fa9-42b2-932b-74966a1eba49","Type":"ContainerDied","Data":"a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53"} Jul 13 15:30:04.496448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.496404 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a068a5834f6361debc0806a92e0a5043d274a39d6e29f3235be2c57505cb9c53" Jul 13 15:30:04.503845 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.498308 2554 generic.go:356] "Generic (PLEG): container finished" podID="a3695726-9bd0-4aa1-a08b-cf0e3fb036d7" containerID="f76852a0364f72db73c7f3265e39b4aea4c45fae0ec174c2302c2d479b79829d" exitCode=0 Jul 13 15:30:04.503845 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.498454 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" event={"ID":"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7","Type":"ContainerDied","Data":"f76852a0364f72db73c7f3265e39b4aea4c45fae0ec174c2302c2d479b79829d"} Jul 13 15:30:04.508398 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.508368 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" event={"ID":"3030602b-edef-4f56-9f73-d50fd834817d","Type":"ContainerDied","Data":"750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635"} Jul 13 15:30:04.508507 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.508426 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750b9bf6f8e5a088a3f09080b26c2b18a031f6601622554a231f262d14866635" Jul 13 15:30:04.508507 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:04.508379 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732610-c4mst" Jul 13 15:30:05.659480 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:05.659453 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:05.732674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:05.732642 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca\") pod \"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7\" (UID: \"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7\") " Jul 13 15:30:05.733055 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:05.733029 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3695726-9bd0-4aa1-a08b-cf0e3fb036d7" (UID: "a3695726-9bd0-4aa1-a08b-cf0e3fb036d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:30:05.834134 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:05.834046 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3695726-9bd0-4aa1-a08b-cf0e3fb036d7-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:06.522497 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:06.522466 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" event={"ID":"a3695726-9bd0-4aa1-a08b-cf0e3fb036d7","Type":"ContainerDied","Data":"27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7"} Jul 13 15:30:06.522497 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:06.522505 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw" Jul 13 15:30:06.522724 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:06.522515 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27fb1eb1ced97296b919e16a554deb033b4689d3c81157c51d08a275784a45a7" Jul 13 15:30:12.648188 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:12.648156 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:12.654205 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:12.654139 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq"] Jul 13 15:30:12.656500 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:12.656478 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-15d63b-29732610-p9jsq"] Jul 13 15:30:13.678897 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:13.678867 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcd203e-3fa9-42b2-932b-74966a1eba49" path="/var/lib/kubelet/pods/bbcd203e-3fa9-42b2-932b-74966a1eba49/volumes" Jul 13 15:30:17.220875 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:17.220835 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:23.045115 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:23.045078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:23.145467 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:23.145428 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:27.906552 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:27.906515 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:28.144302 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.144265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:28.265433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.265402 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56"] Jul 13 15:30:28.265975 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.265950 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcd203e-3fa9-42b2-932b-74966a1eba49" containerName="cleanup" Jul 13 15:30:28.265975 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.265968 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcd203e-3fa9-42b2-932b-74966a1eba49" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.265989 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fae1dd-a9db-465e-9250-759c40c0d175" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.265996 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fae1dd-a9db-465e-9250-759c40c0d175" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266002 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3695726-9bd0-4aa1-a08b-cf0e3fb036d7" containerName="revoke-keys" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266008 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3695726-9bd0-4aa1-a08b-cf0e3fb036d7" containerName="revoke-keys" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266022 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3030602b-edef-4f56-9f73-d50fd834817d" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266027 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3030602b-edef-4f56-9f73-d50fd834817d" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266097 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3030602b-edef-4f56-9f73-d50fd834817d" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266106 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcd203e-3fa9-42b2-932b-74966a1eba49" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266113 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2fae1dd-a9db-465e-9250-759c40c0d175" containerName="cleanup" Jul 13 15:30:28.266149 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.266121 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3695726-9bd0-4aa1-a08b-cf0e3fb036d7" containerName="revoke-keys" Jul 13 15:30:28.269652 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.269625 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:28.272755 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.272730 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:30:28.276557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.276488 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56"] Jul 13 15:30:28.342953 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.342911 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f68651-xmm56\" (UID: \"96df2bcf-cbb4-4239-b7e4-fa80490e50b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:28.443616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.443572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f68651-xmm56\" (UID: \"96df2bcf-cbb4-4239-b7e4-fa80490e50b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:28.444279 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.444258 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f68651-xmm56\" (UID: \"96df2bcf-cbb4-4239-b7e4-fa80490e50b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:28.582317 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.582235 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:28.724613 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.724591 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56"] Jul 13 15:30:28.726764 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:30:28.726731 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96df2bcf_cbb4_4239_b7e4_fa80490e50b0.slice/crio-087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688 WatchSource:0}: Error finding container 087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688: Status 404 returned error can't find the container with id 087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688 Jul 13 15:30:28.856200 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.856115 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:28.953477 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:28.953440 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:29.653560 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:29.653527 2554 generic.go:356] "Generic (PLEG): container finished" podID="96df2bcf-cbb4-4239-b7e4-fa80490e50b0" containerID="b04ee2243936ecd7bf94506218b40f96a7ed7419f27bb4b8544851c294f92b18" exitCode=0 Jul 13 15:30:29.653724 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:29.653580 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" event={"ID":"96df2bcf-cbb4-4239-b7e4-fa80490e50b0","Type":"ContainerDied","Data":"b04ee2243936ecd7bf94506218b40f96a7ed7419f27bb4b8544851c294f92b18"} Jul 13 15:30:29.653724 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:29.653622 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" event={"ID":"96df2bcf-cbb4-4239-b7e4-fa80490e50b0","Type":"ContainerStarted","Data":"087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688"} Jul 13 15:30:30.808135 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:30.808112 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:30.865848 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:30.865819 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca\") pod \"96df2bcf-cbb4-4239-b7e4-fa80490e50b0\" (UID: \"96df2bcf-cbb4-4239-b7e4-fa80490e50b0\") " Jul 13 15:30:30.866178 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:30.866158 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96df2bcf-cbb4-4239-b7e4-fa80490e50b0" (UID: "96df2bcf-cbb4-4239-b7e4-fa80490e50b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:30:30.966818 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:30.966718 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96df2bcf-cbb4-4239-b7e4-fa80490e50b0-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:31.665347 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:31.665313 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" event={"ID":"96df2bcf-cbb4-4239-b7e4-fa80490e50b0","Type":"ContainerDied","Data":"087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688"} Jul 13 15:30:31.665519 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:31.665356 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f68651-xmm56" Jul 13 15:30:31.665519 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:31.665368 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087b9c3cffea73b3ba0a631f2cfaf1aa2b95d1bfccdd90c33ff51bdbcc562688" Jul 13 15:30:38.715628 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:38.715598 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc"] Jul 13 15:30:38.724154 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:38.724126 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f68651-29732610-qd7qc"] Jul 13 15:30:38.730678 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:38.730650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:39.679626 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:39.679593 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2fae1dd-a9db-465e-9250-759c40c0d175" path="/var/lib/kubelet/pods/b2fae1dd-a9db-465e-9250-759c40c0d175/volumes" Jul 13 15:30:43.355921 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:43.355885 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:49.047950 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:49.047918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:49.111501 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:49.111468 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:54.041470 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.041439 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:30:54.041957 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.041651 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" containerID="cri-o://55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188" gracePeriod=30 Jul 13 15:30:54.063912 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.063883 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:54.514357 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.514330 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:54.567339 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.567307 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:54.659745 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.659682 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:54.974119 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:54.974043 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:55.171081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:55.171042 2554 patch_prober.go:28] interesting pod/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 15:30:55.171439 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:55.171107 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 15:30:55.185447 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:55.185415 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:58.275771 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:58.275670 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:58.373570 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:58.373537 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:58.502772 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:58.502730 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:58.718273 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:58.718195 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:59.040222 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.040190 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:59.141057 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.141029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:30:59.309183 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.309158 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:30:59.429577 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429545 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429592 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429630 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429682 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429740 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429821 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429856 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lxx5\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429905 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.429944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429941 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data\") pod \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\" (UID: \"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d\") " Jul 13 15:30:59.430196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429967 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:30:59.430196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.429970 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:30:59.430196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430070 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:30:59.430196 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430118 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:30:59.430395 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430316 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.430395 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430336 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istiod-ca-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.430395 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430351 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-workload-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.430395 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430364 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-credential-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.431010 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.430985 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:30:59.432188 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.432163 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:30:59.432271 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.432220 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:30:59.432383 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.432367 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:30:59.432432 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.432390 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5" (OuterVolumeSpecName: "kube-api-access-7lxx5") pod "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" (UID: "9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d"). InnerVolumeSpecName "kube-api-access-7lxx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:30:59.530923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.530894 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-envoy\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.530923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.530919 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxx5\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-kube-api-access-7lxx5\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.530923 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.530929 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.531134 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.530938 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-data\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.531134 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.530947 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d-istio-podinfo\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:30:59.820382 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.820343 2554 generic.go:356] "Generic (PLEG): container finished" podID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerID="55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188" exitCode=0 Jul 13 15:30:59.820576 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.820397 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" Jul 13 15:30:59.820576 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.820434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerDied","Data":"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188"} Jul 13 15:30:59.820576 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.820503 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd" event={"ID":"9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d","Type":"ContainerDied","Data":"fe10c3d4c7ec9c140ecc1fed3693ae16d9694daf360c5c31c8e3c32ca7c2ed33"} Jul 13 15:30:59.820576 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.820532 2554 scope.go:117] "RemoveContainer" containerID="55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188" Jul 13 15:30:59.833529 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.833503 2554 scope.go:117] "RemoveContainer" containerID="c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36" Jul 13 15:30:59.843202 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.843073 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:30:59.843202 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.843158 2554 scope.go:117] "RemoveContainer" containerID="55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188" Jul 13 15:30:59.843478 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:30:59.843461 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188\": container with ID starting with 55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188 not found: ID does not exist" containerID="55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188" Jul 13 15:30:59.843532 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.843486 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188"} err="failed to get container status \"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188\": rpc error: code = NotFound desc = could not find container \"55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188\": container with ID starting with 55dedce2a2709dee0ca917050614f226e4751c83f07450a0c17e4a2144899188 not found: ID does not exist" Jul 13 15:30:59.843532 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.843509 2554 scope.go:117] "RemoveContainer" containerID="c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36" Jul 13 15:30:59.843998 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:30:59.843893 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36\": container with ID starting with c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36 not found: ID does not exist" containerID="c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36" Jul 13 15:30:59.843998 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.843991 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36"} err="failed to get container status \"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36\": rpc error: code = NotFound desc = could not find container \"c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36\": container with ID starting with c9dc71ed1bb5f7f088d08f26c8e4a1c4a7d553818cecb48e4937aab18d355a36 not found: ID does not exist" Jul 13 15:30:59.846882 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:30:59.846859 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f68651-openshift-default-5d5789b667-7q9rd"] Jul 13 15:31:01.679863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:01.679822 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" path="/var/lib/kubelet/pods/9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d/volumes" Jul 13 15:31:02.820884 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:02.820846 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:03.275952 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.275924 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:03.555946 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.555865 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:31:03.556559 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556542 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.556611 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556564 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.556611 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556583 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96df2bcf-cbb4-4239-b7e4-fa80490e50b0" containerName="revoke-keys" Jul 13 15:31:03.556611 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556592 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="96df2bcf-cbb4-4239-b7e4-fa80490e50b0" containerName="revoke-keys" Jul 13 15:31:03.556611 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556607 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.556744 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556615 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.556744 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556724 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="96df2bcf-cbb4-4239-b7e4-fa80490e50b0" containerName="revoke-keys" Jul 13 15:31:03.556744 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556740 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.556859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.556751 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9182f9b3-7a6f-4f20-913e-e9ed2d0e0a8d" containerName="istio-proxy" Jul 13 15:31:03.561928 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.561904 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.565848 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.565820 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d3da1a-openshift-default-dockercfg-qrxct\"" Jul 13 15:31:03.579148 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.579123 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:31:03.593029 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.592996 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:03.671099 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671064 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671266 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671104 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671266 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671128 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671266 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671153 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671275 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzndz\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671359 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671406 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.671476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.671442 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772410 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772377 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772464 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772495 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772522 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772555 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772812 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772630 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzndz\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772812 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772664 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772812 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772687 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.772812 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.772741 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.773500 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.773474 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.773672 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.773647 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.773749 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.773669 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.773749 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.773708 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.773749 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.773546 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.775345 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.775319 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.775703 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.775678 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.781195 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.781172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.781314 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.781292 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzndz\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz\") pod \"e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:03.842001 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.841915 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:03.881142 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:03.881114 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:04.031375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.031352 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:31:04.033959 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:31:04.033934 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d7cb22_2f96_41b7_8187_eb84df310e2b.slice/crio-537b937916ce308fa65328df32215b4904547ace94ee6f907a45fa652e217889 WatchSource:0}: Error finding container 537b937916ce308fa65328df32215b4904547ace94ee6f907a45fa652e217889: Status 404 returned error can't find the container with id 537b937916ce308fa65328df32215b4904547ace94ee6f907a45fa652e217889 Jul 13 15:31:04.036295 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.036255 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:04.036378 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.036343 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:04.036418 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.036386 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:04.851961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.851925 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerStarted","Data":"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038"} Jul 13 15:31:04.852360 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.851982 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerStarted","Data":"537b937916ce308fa65328df32215b4904547ace94ee6f907a45fa652e217889"} Jul 13 15:31:04.872808 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.872746 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podStartSLOduration=1.872729719 podStartE2EDuration="1.872729719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:31:04.871321146 +0000 UTC m=+2327.761996285" watchObservedRunningTime="2026-07-13 15:31:04.872729719 +0000 UTC m=+2327.763404820" Jul 13 15:31:04.881367 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:04.881344 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:05.886748 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:05.886721 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:06.817509 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:06.817463 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:06.863817 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:06.863760 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:06.864545 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:06.864527 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:06.902568 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:06.902525 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:07.021294 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:07.021248 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:07.590107 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:07.590071 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:08.923712 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:08.923681 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:11.388429 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:11.388349 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:19.477459 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.477424 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk"] Jul 13 15:31:19.482301 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.482282 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:19.484931 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.484902 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:31:19.489268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.489243 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk"] Jul 13 15:31:19.518945 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.518906 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b866e-62pwk\" (UID: \"c023008b-29be-4295-9b3d-aa272edcb870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:19.620376 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.620336 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b866e-62pwk\" (UID: \"c023008b-29be-4295-9b3d-aa272edcb870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:19.621460 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.621432 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b866e-62pwk\" (UID: \"c023008b-29be-4295-9b3d-aa272edcb870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:19.793379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.793347 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:19.808993 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.808965 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:19.911944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:19.911735 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:20.015495 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:20.015472 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk"] Jul 13 15:31:20.017875 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:31:20.017841 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc023008b_29be_4295_9b3d_aa272edcb870.slice/crio-0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9 WatchSource:0}: Error finding container 0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9: Status 404 returned error can't find the container with id 0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9 Jul 13 15:31:20.970170 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:20.970136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" event={"ID":"c023008b-29be-4295-9b3d-aa272edcb870","Type":"ContainerStarted","Data":"72b517861acf6662355938f560f36154226ebd3b92fe8b1c05645947c4d4974e"} Jul 13 15:31:20.970547 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:20.970197 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" event={"ID":"c023008b-29be-4295-9b3d-aa272edcb870","Type":"ContainerStarted","Data":"0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9"} Jul 13 15:31:20.988730 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:20.988681 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" podStartSLOduration=1.9886660250000001 podStartE2EDuration="1.988666025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:31:20.98680075 +0000 UTC m=+2343.877475843" watchObservedRunningTime="2026-07-13 15:31:20.988666025 +0000 UTC m=+2343.879341127" Jul 13 15:31:21.976952 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:21.976878 2554 generic.go:356] "Generic (PLEG): container finished" podID="c023008b-29be-4295-9b3d-aa272edcb870" containerID="72b517861acf6662355938f560f36154226ebd3b92fe8b1c05645947c4d4974e" exitCode=0 Jul 13 15:31:21.977298 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:21.976951 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" event={"ID":"c023008b-29be-4295-9b3d-aa272edcb870","Type":"ContainerDied","Data":"72b517861acf6662355938f560f36154226ebd3b92fe8b1c05645947c4d4974e"} Jul 13 15:31:22.904969 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:22.904935 2554 patch_prober.go:28] interesting pod/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 15:31:22.905158 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:22.904990 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 15:31:23.142376 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.142351 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:23.164498 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.164427 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca\") pod \"c023008b-29be-4295-9b3d-aa272edcb870\" (UID: \"c023008b-29be-4295-9b3d-aa272edcb870\") " Jul 13 15:31:23.164884 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.164858 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c023008b-29be-4295-9b3d-aa272edcb870" (UID: "c023008b-29be-4295-9b3d-aa272edcb870"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:31:23.265333 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.265301 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c023008b-29be-4295-9b3d-aa272edcb870-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:31:23.990232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.990160 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" Jul 13 15:31:23.990232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.990173 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b866e-62pwk" event={"ID":"c023008b-29be-4295-9b3d-aa272edcb870","Type":"ContainerDied","Data":"0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9"} Jul 13 15:31:23.990232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:23.990217 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d68b68ca13428d9015ce92a628ffe55a0b0e5e32a9a70228f7130075a6cfff9" Jul 13 15:31:24.997129 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.997086 2554 generic.go:356] "Generic (PLEG): container finished" podID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerID="d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038" exitCode=0 Jul 13 15:31:24.997575 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.997154 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerDied","Data":"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038"} Jul 13 15:31:24.997575 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.997558 2554 scope.go:117] "RemoveContainer" containerID="d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038" Jul 13 15:31:24.998524 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.998499 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:24.998592 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.998565 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:24.998634 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:24.998596 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:31:26.004144 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:26.004106 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerStarted","Data":"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27"} Jul 13 15:31:26.882054 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:26.882019 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:26.882268 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:26.882248 2554 patch_prober.go:28] interesting pod/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 13 15:31:26.882332 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:26.882292 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 13 15:31:27.882264 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:27.882226 2554 patch_prober.go:28] interesting pod/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 13 15:31:27.882634 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:27.882294 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 13 15:31:28.881525 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:28.881486 2554 patch_prober.go:28] interesting pod/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Jul 13 15:31:28.881703 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:28.881557 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Jul 13 15:31:29.345969 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.345924 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:29.446655 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.446610 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:29.647867 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.646840 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:29.937030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.936945 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:29.937449 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.937417 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:29.938556 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:29.938535 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:31:34.559408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:34.559374 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:40.446332 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:40.446299 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:40.546207 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:40.546177 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:45.259571 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.259541 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:45.496318 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.496283 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:45.608088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.608000 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt"] Jul 13 15:31:45.608467 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.608452 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c023008b-29be-4295-9b3d-aa272edcb870" containerName="revoke-keys" Jul 13 15:31:45.608467 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.608466 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c023008b-29be-4295-9b3d-aa272edcb870" containerName="revoke-keys" Jul 13 15:31:45.608601 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.608551 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c023008b-29be-4295-9b3d-aa272edcb870" containerName="revoke-keys" Jul 13 15:31:45.610700 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.610680 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:45.613492 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.613472 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:31:45.619728 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.619704 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt"] Jul 13 15:31:45.691077 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.691049 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt\" (UID: \"7d586b70-1493-4485-8b51-0e479ecfe1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:45.791976 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.791945 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt\" (UID: \"7d586b70-1493-4485-8b51-0e479ecfe1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:45.792637 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.792615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt\" (UID: \"7d586b70-1493-4485-8b51-0e479ecfe1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:45.922910 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:45.922814 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:46.073597 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:46.073572 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt"] Jul 13 15:31:46.075601 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:31:46.075574 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d586b70_1493_4485_8b51_0e479ecfe1a1.slice/crio-da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5 WatchSource:0}: Error finding container da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5: Status 404 returned error can't find the container with id da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5 Jul 13 15:31:46.127341 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:46.127312 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" event={"ID":"7d586b70-1493-4485-8b51-0e479ecfe1a1","Type":"ContainerStarted","Data":"da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5"} Jul 13 15:31:47.134014 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:47.133934 2554 generic.go:356] "Generic (PLEG): container finished" podID="7d586b70-1493-4485-8b51-0e479ecfe1a1" containerID="cc4e8913c1f2bd5698864dcbcab6835d8efb62507ff6ca5ba43a0168e2049230" exitCode=0 Jul 13 15:31:47.134375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:47.134019 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" event={"ID":"7d586b70-1493-4485-8b51-0e479ecfe1a1","Type":"ContainerDied","Data":"cc4e8913c1f2bd5698864dcbcab6835d8efb62507ff6ca5ba43a0168e2049230"} Jul 13 15:31:48.290380 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:48.290351 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:48.418313 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:48.418211 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca\") pod \"7d586b70-1493-4485-8b51-0e479ecfe1a1\" (UID: \"7d586b70-1493-4485-8b51-0e479ecfe1a1\") " Jul 13 15:31:48.418559 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:48.418535 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d586b70-1493-4485-8b51-0e479ecfe1a1" (UID: "7d586b70-1493-4485-8b51-0e479ecfe1a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:31:48.519620 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:48.519569 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d586b70-1493-4485-8b51-0e479ecfe1a1-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:31:49.146583 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:49.146555 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" Jul 13 15:31:49.146752 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:49.146595 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt" event={"ID":"7d586b70-1493-4485-8b51-0e479ecfe1a1","Type":"ContainerDied","Data":"da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5"} Jul 13 15:31:49.146752 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:49.146640 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7f2592ca74f0d563d79bdc97356f4728d376732b19963e517ee3018093b3b5" Jul 13 15:31:51.697192 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:51.697160 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:31:56.016901 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:31:56.016864 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:00.700258 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:00.700220 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:06.169914 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:06.169881 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:06.263569 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:06.263525 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:06.776145 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:06.776110 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:06.834808 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:06.834767 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:11.383021 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:11.382987 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:32:11.383504 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:11.383248 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" containerID="cri-o://a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27" gracePeriod=30 Jul 13 15:32:11.402079 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:11.402052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:13.771380 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:13.771343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:14.939751 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:14.939715 2554 patch_prober.go:28] interesting pod/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 15:32:14.940143 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:14.939771 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 15:32:16.470406 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.470371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:16.578407 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.578312 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:16.648373 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.648312 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:32:16.785060 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785024 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785263 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785127 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785263 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785169 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785263 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785227 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785263 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785258 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785285 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785294 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:32:16.785486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785314 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzndz\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785349 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785486 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785397 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket\") pod \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\" (UID: \"c6d7cb22-2f96-41b7-8187-eb84df310e2b\") " Jul 13 15:32:16.785730 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785567 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:32:16.785821 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785757 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:32:16.785821 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785779 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.785918 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.785832 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-credential-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.786037 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.786012 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:32:16.786308 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.786285 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:32:16.787502 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.787469 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:32:16.787611 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.787569 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz" (OuterVolumeSpecName: "kube-api-access-rzndz") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "kube-api-access-rzndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:32:16.787798 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.787759 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:32:16.787798 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.787772 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6d7cb22-2f96-41b7-8187-eb84df310e2b" (UID: "c6d7cb22-2f96-41b7-8187-eb84df310e2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:32:16.887141 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887108 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-podinfo\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887141 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887136 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-envoy\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887141 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887145 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887152 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istio-data\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887161 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzndz\" (UniqueName: \"kubernetes.io/projected/c6d7cb22-2f96-41b7-8187-eb84df310e2b-kube-api-access-rzndz\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887172 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d7cb22-2f96-41b7-8187-eb84df310e2b-istiod-ca-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:16.887375 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:16.887180 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d7cb22-2f96-41b7-8187-eb84df310e2b-workload-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:32:17.316835 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.316804 2554 generic.go:356] "Generic (PLEG): container finished" podID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerID="a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27" exitCode=0 Jul 13 15:32:17.317010 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.316869 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" Jul 13 15:32:17.317010 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.316872 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerDied","Data":"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27"} Jul 13 15:32:17.317010 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.316987 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr" event={"ID":"c6d7cb22-2f96-41b7-8187-eb84df310e2b","Type":"ContainerDied","Data":"537b937916ce308fa65328df32215b4904547ace94ee6f907a45fa652e217889"} Jul 13 15:32:17.317010 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.317010 2554 scope.go:117] "RemoveContainer" containerID="a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27" Jul 13 15:32:17.330056 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.329992 2554 scope.go:117] "RemoveContainer" containerID="d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038" Jul 13 15:32:17.338603 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.338558 2554 scope.go:117] "RemoveContainer" containerID="a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27" Jul 13 15:32:17.338891 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:32:17.338864 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27\": container with ID starting with a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27 not found: ID does not exist" containerID="a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27" Jul 13 15:32:17.338987 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.338903 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27"} err="failed to get container status \"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27\": rpc error: code = NotFound desc = could not find container \"a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27\": container with ID starting with a4fe5c5c50ed799fd1d8d29b9bde1ced4f29f6729d34ef8dbc888bedbb824d27 not found: ID does not exist" Jul 13 15:32:17.338987 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.338921 2554 scope.go:117] "RemoveContainer" containerID="d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038" Jul 13 15:32:17.339211 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:32:17.339193 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038\": container with ID starting with d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038 not found: ID does not exist" containerID="d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038" Jul 13 15:32:17.339257 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.339217 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038"} err="failed to get container status \"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038\": rpc error: code = NotFound desc = could not find container \"d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038\": container with ID starting with d7dd91fb6c61bbc563c583603ab17bc9ebcb5a9be6dffd517b79690fa7617038 not found: ID does not exist" Jul 13 15:32:17.341052 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.341032 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:32:17.344778 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.344760 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d3da1a-openshift-default-f65666dcc-j4hzr"] Jul 13 15:32:17.680365 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.680284 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" path="/var/lib/kubelet/pods/c6d7cb22-2f96-41b7-8187-eb84df310e2b/volumes" Jul 13 15:32:17.775342 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.775314 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:32:17.781154 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.781128 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:32:17.785885 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.785866 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:32:17.791325 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:17.791308 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:32:20.193677 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.193637 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:20.537841 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.537779 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:20.818293 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818206 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818899 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d586b70-1493-4485-8b51-0e479ecfe1a1" containerName="revoke-keys" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818923 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d586b70-1493-4485-8b51-0e479ecfe1a1" containerName="revoke-keys" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818946 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818955 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818968 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.818977 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.819088 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d586b70-1493-4485-8b51-0e479ecfe1a1" containerName="revoke-keys" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.819099 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.820092 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.819111 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d7cb22-2f96-41b7-8187-eb84df310e2b" containerName="istio-proxy" Jul 13 15:32:20.824433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.824410 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.827087 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.827069 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-450dce-openshift-default-dockercfg-h4w7x\"" Jul 13 15:32:20.832372 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.832349 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:32:20.845334 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.845309 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:20.926918 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.926881 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.926942 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927035 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5cf\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927141 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927189 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:20.927363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:20.927268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028475 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028475 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028475 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028614 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028669 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5cf\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.028733 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028698 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029067 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028740 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029067 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.028993 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029067 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.029025 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029374 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.029343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.029381 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.029505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.029396 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.031266 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.031237 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.031450 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.031433 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.038037 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.038016 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.038148 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.038051 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5cf\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf\") pod \"e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.138203 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.138102 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:21.248448 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.248406 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:21.289053 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.289022 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:32:21.290125 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:32:21.290097 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb214ba35_5b22_4dd8_b1ef_90f60b2d8979.slice/crio-a9c57e2a08340eb7baa5e60bee184b277c8b5c003bad9c4d802d6cbddddf8936 WatchSource:0}: Error finding container a9c57e2a08340eb7baa5e60bee184b277c8b5c003bad9c4d802d6cbddddf8936: Status 404 returned error can't find the container with id a9c57e2a08340eb7baa5e60bee184b277c8b5c003bad9c4d802d6cbddddf8936 Jul 13 15:32:21.292261 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.292231 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:21.292327 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.292293 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:21.292327 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.292323 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:21.340304 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:21.340259 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerStarted","Data":"a9c57e2a08340eb7baa5e60bee184b277c8b5c003bad9c4d802d6cbddddf8936"} Jul 13 15:32:22.348886 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:22.348847 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerStarted","Data":"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796"} Jul 13 15:32:22.374178 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:22.374129 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podStartSLOduration=2.374113084 podStartE2EDuration="2.374113084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:32:22.372950393 +0000 UTC m=+2405.263625494" watchObservedRunningTime="2026-07-13 15:32:22.374113084 +0000 UTC m=+2405.264788186" Jul 13 15:32:23.138972 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:23.138931 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:23.144083 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:23.144060 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:23.359870 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:23.359840 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:23.360751 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:23.360729 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:24.321815 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:24.321751 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:24.398256 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:24.398222 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:24.475882 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:24.475834 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:24.577138 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:24.577031 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:24.961078 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:24.960999 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:25.061643 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:25.061600 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:28.817743 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:28.817711 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:37.034359 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:37.034147 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:37.134725 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:37.134699 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:37.235581 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:37.235548 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:39.400970 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:39.400929 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 15:32:39.401363 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:39.401001 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 15:32:40.465111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.465018 2554 generic.go:356] "Generic (PLEG): container finished" podID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerID="a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796" exitCode=0 Jul 13 15:32:40.465111 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.465084 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerDied","Data":"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796"} Jul 13 15:32:40.465537 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.465492 2554 scope.go:117] "RemoveContainer" containerID="a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796" Jul 13 15:32:40.466533 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.466507 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:40.466616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.466579 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:40.466616 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:40.466611 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 13 15:32:41.472275 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:41.472242 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerStarted","Data":"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8"} Jul 13 15:32:42.139276 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:42.139237 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:42.139557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:42.139426 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Jul 13 15:32:42.139557 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:42.139471 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Jul 13 15:32:43.139255 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:43.139213 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Jul 13 15:32:43.139754 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:43.139303 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Jul 13 15:32:44.138533 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:44.138492 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Jul 13 15:32:44.138743 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:44.138560 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Jul 13 15:32:45.139140 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:45.139091 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Jul 13 15:32:45.139634 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:45.139173 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Jul 13 15:32:46.139386 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:46.139338 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Jul 13 15:32:46.139904 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:46.139422 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Jul 13 15:32:46.744605 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:46.744572 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:46.878025 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:46.877997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:32:47.144393 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:47.144365 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:47.144849 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:47.144831 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:32:47.145487 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:32:47.145470 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:33:08.422827 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:33:08.422775 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:12.700311 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:12.700209 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:12.801460 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:12.801410 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:24.741884 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:24.741839 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:24.846623 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:24.846591 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:45.385914 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:45.385865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:34:45.472640 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:34:45.472588 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:35:04.512322 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:35:04.512287 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:35:17.865295 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:35:17.865257 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:35:18.842176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:35:18.842145 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:35:41.890895 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:35:41.890812 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:36:08.638402 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:36:08.638365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:36:17.989181 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:36:17.989151 2554 scope.go:117] "RemoveContainer" containerID="268f504d83e9ded7baecb33dc36157bd9ea18036df023da43b044c9c9a90a8aa" Jul 13 15:36:17.998177 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:36:17.998154 2554 scope.go:117] "RemoveContainer" containerID="e809ed1ea6d78f37d8ff6845799330661aa14d6a46e72959ed891f64a4d834a4" Jul 13 15:37:01.142995 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:01.142961 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:02.077539 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:02.077500 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:06.292539 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:06.292486 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:07.018505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:07.018452 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:07.125892 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:07.125846 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:17.810540 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:17.810452 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:37:17.816694 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:17.816672 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:37:17.823859 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:17.823838 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:37:17.831353 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:17.831333 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-khk5x_a425f617-f69e-48d4-9995-52d948d5c86e/ovn-acl-logging/0.log" Jul 13 15:37:24.477737 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:24.477705 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:37.994680 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:37.994643 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:38.130095 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:38.130061 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:46.412831 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:46.412770 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:47.530872 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:47.530832 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:37:47.628310 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:37:47.628244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:38:29.190104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:29.190068 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:38:30.012353 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:30.012321 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:38:31.009843 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.009805 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682981584048"] Jul 13 15:38:31.014036 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.014018 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:31.017012 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.016984 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-27xqh\"" Jul 13 15:38:31.019304 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.019282 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682981584048"] Jul 13 15:38:31.129416 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.129384 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2sh\" (UniqueName: \"kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh\") pod \"test-curl-7715-140682981584048\" (UID: \"e1e10086-bb88-4362-88d6-ed4bbf3f8251\") " pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:31.230773 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.230730 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2sh\" (UniqueName: \"kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh\") pod \"test-curl-7715-140682981584048\" (UID: \"e1e10086-bb88-4362-88d6-ed4bbf3f8251\") " pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:31.240118 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.240088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2sh\" (UniqueName: \"kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh\") pod \"test-curl-7715-140682981584048\" (UID: \"e1e10086-bb88-4362-88d6-ed4bbf3f8251\") " pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:31.326751 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.326677 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:31.472901 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.472878 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682981584048"] Jul 13 15:38:31.475626 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:31.475600 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1e10086_bb88_4362_88d6_ed4bbf3f8251.slice/crio-f1270675531a693f3cb8a78e1a5cd3bcce3ca32ffd44824c292644cf98f2639e WatchSource:0}: Error finding container f1270675531a693f3cb8a78e1a5cd3bcce3ca32ffd44824c292644cf98f2639e: Status 404 returned error can't find the container with id f1270675531a693f3cb8a78e1a5cd3bcce3ca32ffd44824c292644cf98f2639e Jul 13 15:38:31.476885 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:31.476865 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 15:38:32.470397 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:32.470371 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682981584048" event={"ID":"e1e10086-bb88-4362-88d6-ed4bbf3f8251","Type":"ContainerStarted","Data":"f1270675531a693f3cb8a78e1a5cd3bcce3ca32ffd44824c292644cf98f2639e"} Jul 13 15:38:33.479468 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:33.479435 2554 generic.go:356] "Generic (PLEG): container finished" podID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" containerID="a2993d3966b95013af1d2f26a7ab5db80bc2fa39bb6bf4e37384ddea261cdab2" exitCode=0 Jul 13 15:38:33.479870 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:33.479521 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682981584048" event={"ID":"e1e10086-bb88-4362-88d6-ed4bbf3f8251","Type":"ContainerDied","Data":"a2993d3966b95013af1d2f26a7ab5db80bc2fa39bb6bf4e37384ddea261cdab2"} Jul 13 15:38:34.634660 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.634631 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:34.655315 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.655285 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682981584048_e1e10086-bb88-4362-88d6-ed4bbf3f8251/test-curl-7715-140682981584048/0.log" Jul 13 15:38:34.681019 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.680996 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682981584048"] Jul 13 15:38:34.683253 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.683232 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682981584048"] Jul 13 15:38:34.764453 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.764428 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk2sh\" (UniqueName: \"kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh\") pod \"e1e10086-bb88-4362-88d6-ed4bbf3f8251\" (UID: \"e1e10086-bb88-4362-88d6-ed4bbf3f8251\") " Jul 13 15:38:34.766481 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.766456 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh" (OuterVolumeSpecName: "kube-api-access-mk2sh") pod "e1e10086-bb88-4362-88d6-ed4bbf3f8251" (UID: "e1e10086-bb88-4362-88d6-ed4bbf3f8251"). InnerVolumeSpecName "kube-api-access-mk2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:38:34.804394 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.804369 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682983027376"] Jul 13 15:38:34.804798 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.804775 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" containerName="test-curl-7715-140682981584048" Jul 13 15:38:34.804856 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.804799 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" containerName="test-curl-7715-140682981584048" Jul 13 15:38:34.804893 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.804881 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" containerName="test-curl-7715-140682981584048" Jul 13 15:38:34.808003 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.807988 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:34.815863 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.815842 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983027376"] Jul 13 15:38:34.865996 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.865969 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2sh\" (UniqueName: \"kubernetes.io/projected/e1e10086-bb88-4362-88d6-ed4bbf3f8251-kube-api-access-mk2sh\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:34.966869 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:34.966823 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrzs\" (UniqueName: \"kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs\") pod \"test-curl-7715-140682983027376\" (UID: \"251f6304-b555-483b-8ebf-5ead93e2b8d9\") " pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:35.068021 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.067948 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrzs\" (UniqueName: \"kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs\") pod \"test-curl-7715-140682983027376\" (UID: \"251f6304-b555-483b-8ebf-5ead93e2b8d9\") " pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:35.078440 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.078415 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrzs\" (UniqueName: \"kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs\") pod \"test-curl-7715-140682983027376\" (UID: \"251f6304-b555-483b-8ebf-5ead93e2b8d9\") " pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:35.117544 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.117515 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:35.258806 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.258759 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983027376"] Jul 13 15:38:35.260501 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:35.260475 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod251f6304_b555_483b_8ebf_5ead93e2b8d9.slice/crio-db7df6c5173b6165f5efd656683c253276ad72d2f437af7de926d8e69f505549 WatchSource:0}: Error finding container db7df6c5173b6165f5efd656683c253276ad72d2f437af7de926d8e69f505549: Status 404 returned error can't find the container with id db7df6c5173b6165f5efd656683c253276ad72d2f437af7de926d8e69f505549 Jul 13 15:38:35.490845 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.490744 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983027376" event={"ID":"251f6304-b555-483b-8ebf-5ead93e2b8d9","Type":"ContainerStarted","Data":"db7df6c5173b6165f5efd656683c253276ad72d2f437af7de926d8e69f505549"} Jul 13 15:38:35.491851 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.491834 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1270675531a693f3cb8a78e1a5cd3bcce3ca32ffd44824c292644cf98f2639e" Jul 13 15:38:35.491971 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.491856 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:35.502599 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:35.502573 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682981584048\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" pod="openshift-ingress/test-curl-7715-140682981584048" Jul 13 15:38:35.681081 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:35.681049 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e10086-bb88-4362-88d6-ed4bbf3f8251" path="/var/lib/kubelet/pods/e1e10086-bb88-4362-88d6-ed4bbf3f8251/volumes" Jul 13 15:38:36.499895 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:36.499862 2554 generic.go:356] "Generic (PLEG): container finished" podID="251f6304-b555-483b-8ebf-5ead93e2b8d9" containerID="706ec030ff4a94752c3d55bae6ab3671830091f37959767746f9a1b1d663cb4c" exitCode=0 Jul 13 15:38:36.500069 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:36.499921 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983027376" event={"ID":"251f6304-b555-483b-8ebf-5ead93e2b8d9","Type":"ContainerDied","Data":"706ec030ff4a94752c3d55bae6ab3671830091f37959767746f9a1b1d663cb4c"} Jul 13 15:38:37.662064 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.662043 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:37.684927 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.684899 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682983027376_251f6304-b555-483b-8ebf-5ead93e2b8d9/test-curl-7715-140682983027376/0.log" Jul 13 15:38:37.709287 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.709256 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682983027376"] Jul 13 15:38:37.717599 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.717471 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682983027376"] Jul 13 15:38:37.794724 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.794692 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rrzs\" (UniqueName: \"kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs\") pod \"251f6304-b555-483b-8ebf-5ead93e2b8d9\" (UID: \"251f6304-b555-483b-8ebf-5ead93e2b8d9\") " Jul 13 15:38:37.796773 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.796744 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs" (OuterVolumeSpecName: "kube-api-access-5rrzs") pod "251f6304-b555-483b-8ebf-5ead93e2b8d9" (UID: "251f6304-b555-483b-8ebf-5ead93e2b8d9"). InnerVolumeSpecName "kube-api-access-5rrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:38:37.892547 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.892513 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:37.893009 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.892996 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" containerName="test-curl-7715-140682983027376" Jul 13 15:38:37.893065 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.893011 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" containerName="test-curl-7715-140682983027376" Jul 13 15:38:37.893110 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.893077 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" containerName="test-curl-7715-140682983027376" Jul 13 15:38:37.895488 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.895468 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rrzs\" (UniqueName: \"kubernetes.io/projected/251f6304-b555-483b-8ebf-5ead93e2b8d9-kube-api-access-5rrzs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:37.896124 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.896109 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:37.905258 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.905234 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:37.996089 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:37.996058 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ld8s\" (UniqueName: \"kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s\") pod \"test-curl-7715-140682983773936\" (UID: \"71a31270-a8c2-4561-8474-6950657fa025\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:38.097103 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.097021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ld8s\" (UniqueName: \"kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s\") pod \"test-curl-7715-140682983773936\" (UID: \"71a31270-a8c2-4561-8474-6950657fa025\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:38.105856 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.105833 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ld8s\" (UniqueName: \"kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s\") pod \"test-curl-7715-140682983773936\" (UID: \"71a31270-a8c2-4561-8474-6950657fa025\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:38.206492 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.206460 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:38.348720 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.348699 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:38.351419 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:38.351388 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a31270_a8c2_4561_8474_6950657fa025.slice/crio-8c6d24345f57582004485556cb7b13fa29ee6dc9593d1ab34a5a3531684ebfe1 WatchSource:0}: Error finding container 8c6d24345f57582004485556cb7b13fa29ee6dc9593d1ab34a5a3531684ebfe1: Status 404 returned error can't find the container with id 8c6d24345f57582004485556cb7b13fa29ee6dc9593d1ab34a5a3531684ebfe1 Jul 13 15:38:38.511831 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.511762 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983773936" event={"ID":"71a31270-a8c2-4561-8474-6950657fa025","Type":"ContainerStarted","Data":"8c6d24345f57582004485556cb7b13fa29ee6dc9593d1ab34a5a3531684ebfe1"} Jul 13 15:38:38.513045 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.513024 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:38.513189 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:38.513050 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7df6c5173b6165f5efd656683c253276ad72d2f437af7de926d8e69f505549" Jul 13 15:38:38.524079 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:38.524050 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682983027376\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:39.520433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:39.520397 2554 generic.go:356] "Generic (PLEG): container finished" podID="71a31270-a8c2-4561-8474-6950657fa025" containerID="52b14aa1a99d74c6ca1ab847c43538fecbbc837282056f2aae592751bb05d473" exitCode=0 Jul 13 15:38:39.520900 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:39.520449 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983773936" event={"ID":"71a31270-a8c2-4561-8474-6950657fa025","Type":"ContainerDied","Data":"52b14aa1a99d74c6ca1ab847c43538fecbbc837282056f2aae592751bb05d473"} Jul 13 15:38:39.538934 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:39.538890 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682983027376\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" pod="openshift-ingress/test-curl-7715-140682983027376" Jul 13 15:38:39.679876 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:39.679846 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="251f6304-b555-483b-8ebf-5ead93e2b8d9" path="/var/lib/kubelet/pods/251f6304-b555-483b-8ebf-5ead93e2b8d9/volumes" Jul 13 15:38:40.671673 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.671644 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:40.693291 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.693263 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682983773936_71a31270-a8c2-4561-8474-6950657fa025/test-curl-7715-140682983773936/0.log" Jul 13 15:38:40.717411 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.717384 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ld8s\" (UniqueName: \"kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s\") pod \"71a31270-a8c2-4561-8474-6950657fa025\" (UID: \"71a31270-a8c2-4561-8474-6950657fa025\") " Jul 13 15:38:40.719505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.719475 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s" (OuterVolumeSpecName: "kube-api-access-8ld8s") pod "71a31270-a8c2-4561-8474-6950657fa025" (UID: "71a31270-a8c2-4561-8474-6950657fa025"). InnerVolumeSpecName "kube-api-access-8ld8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:38:40.719958 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.719936 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:40.724606 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.724584 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:40.819013 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.818939 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ld8s\" (UniqueName: \"kubernetes.io/projected/71a31270-a8c2-4561-8474-6950657fa025-kube-api-access-8ld8s\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:40.852403 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.852371 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:40.852819 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.852806 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a31270-a8c2-4561-8474-6950657fa025" containerName="test-curl-7715-140682983773936" Jul 13 15:38:40.852860 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.852820 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a31270-a8c2-4561-8474-6950657fa025" containerName="test-curl-7715-140682983773936" Jul 13 15:38:40.852912 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.852896 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a31270-a8c2-4561-8474-6950657fa025" containerName="test-curl-7715-140682983773936" Jul 13 15:38:40.855927 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.855911 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:40.861799 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.861763 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:40.919488 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:40.919460 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzfp\" (UniqueName: \"kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp\") pod \"test-curl-7715-140682983773936\" (UID: \"f0777bc7-6c92-4bc6-a898-40e0fb402f5b\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:41.020067 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.020033 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzfp\" (UniqueName: \"kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp\") pod \"test-curl-7715-140682983773936\" (UID: \"f0777bc7-6c92-4bc6-a898-40e0fb402f5b\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:41.028727 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.028698 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzfp\" (UniqueName: \"kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp\") pod \"test-curl-7715-140682983773936\" (UID: \"f0777bc7-6c92-4bc6-a898-40e0fb402f5b\") " pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:41.166977 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.166887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:41.304890 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.304862 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:41.306538 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:41.306512 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0777bc7_6c92_4bc6_a898_40e0fb402f5b.slice/crio-847aed5fa8ab7492f85361c536d4325e330350220c6dff365a341ea4c815dd51 WatchSource:0}: Error finding container 847aed5fa8ab7492f85361c536d4325e330350220c6dff365a341ea4c815dd51: Status 404 returned error can't find the container with id 847aed5fa8ab7492f85361c536d4325e330350220c6dff365a341ea4c815dd51 Jul 13 15:38:41.532830 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.532776 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6d24345f57582004485556cb7b13fa29ee6dc9593d1ab34a5a3531684ebfe1" Jul 13 15:38:41.532830 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.532780 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:41.534000 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.533969 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983773936" event={"ID":"f0777bc7-6c92-4bc6-a898-40e0fb402f5b","Type":"ContainerStarted","Data":"847aed5fa8ab7492f85361c536d4325e330350220c6dff365a341ea4c815dd51"} Jul 13 15:38:41.545044 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.545016 2554 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7715-140682983773936" oldPodUID="71a31270-a8c2-4561-8474-6950657fa025" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" Jul 13 15:38:41.680823 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:41.680776 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a31270-a8c2-4561-8474-6950657fa025" path="/var/lib/kubelet/pods/71a31270-a8c2-4561-8474-6950657fa025/volumes" Jul 13 15:38:42.542360 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:42.542322 2554 generic.go:356] "Generic (PLEG): container finished" podID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" containerID="8d0e6173063a45433b7063a75cef9aab5bf7f1e78cfbdf75b2cf07b53f00fc96" exitCode=0 Jul 13 15:38:42.542533 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:42.542403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682983773936" event={"ID":"f0777bc7-6c92-4bc6-a898-40e0fb402f5b","Type":"ContainerDied","Data":"8d0e6173063a45433b7063a75cef9aab5bf7f1e78cfbdf75b2cf07b53f00fc96"} Jul 13 15:38:43.696338 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.696314 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:43.721153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.721125 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682983773936_f0777bc7-6c92-4bc6-a898-40e0fb402f5b/test-curl-7715-140682983773936/0.log" Jul 13 15:38:43.746661 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.746636 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttzfp\" (UniqueName: \"kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp\") pod \"f0777bc7-6c92-4bc6-a898-40e0fb402f5b\" (UID: \"f0777bc7-6c92-4bc6-a898-40e0fb402f5b\") " Jul 13 15:38:43.748145 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.748120 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:43.748661 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.748641 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp" (OuterVolumeSpecName: "kube-api-access-ttzfp") pod "f0777bc7-6c92-4bc6-a898-40e0fb402f5b" (UID: "f0777bc7-6c92-4bc6-a898-40e0fb402f5b"). InnerVolumeSpecName "kube-api-access-ttzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:38:43.751872 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.751852 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682983773936"] Jul 13 15:38:43.848356 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.848274 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzfp\" (UniqueName: \"kubernetes.io/projected/f0777bc7-6c92-4bc6-a898-40e0fb402f5b-kube-api-access-ttzfp\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:43.883294 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.883256 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"] Jul 13 15:38:43.883716 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.883702 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" containerName="test-curl-7715-140682983773936" Jul 13 15:38:43.883755 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.883718 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" containerName="test-curl-7715-140682983773936" Jul 13 15:38:43.883826 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.883816 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" containerName="test-curl-7715-140682983773936" Jul 13 15:38:43.887015 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.886999 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:43.895841 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.895818 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"] Jul 13 15:38:43.949325 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:43.949284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99h2\" (UniqueName: \"kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2\") pod \"test-curl-7715-140682982229312\" (UID: \"e10b68c1-bf89-4fc6-b3f2-484c76c569cf\") " pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:44.050036 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.049993 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99h2\" (UniqueName: \"kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2\") pod \"test-curl-7715-140682982229312\" (UID: \"e10b68c1-bf89-4fc6-b3f2-484c76c569cf\") " pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:44.059136 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.059109 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99h2\" (UniqueName: \"kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2\") pod \"test-curl-7715-140682982229312\" (UID: \"e10b68c1-bf89-4fc6-b3f2-484c76c569cf\") " pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:44.196172 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.196100 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:44.349027 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.349004 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"] Jul 13 15:38:44.351113 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:44.351088 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10b68c1_bf89_4fc6_b3f2_484c76c569cf.slice/crio-32ff5af82dcb42d7a95f8c8cbbc50814e567090d16896436d2d80d0c258feb85 WatchSource:0}: Error finding container 32ff5af82dcb42d7a95f8c8cbbc50814e567090d16896436d2d80d0c258feb85: Status 404 returned error can't find the container with id 32ff5af82dcb42d7a95f8c8cbbc50814e567090d16896436d2d80d0c258feb85 Jul 13 15:38:44.554697 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.554671 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:44.554902 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.554712 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847aed5fa8ab7492f85361c536d4325e330350220c6dff365a341ea4c815dd51" Jul 13 15:38:44.555973 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:44.555944 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682982229312" event={"ID":"e10b68c1-bf89-4fc6-b3f2-484c76c569cf","Type":"ContainerStarted","Data":"32ff5af82dcb42d7a95f8c8cbbc50814e567090d16896436d2d80d0c258feb85"} Jul 13 15:38:44.565749 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:44.565718 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682983773936\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:45.563032 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:45.562994 2554 generic.go:356] "Generic (PLEG): container finished" podID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" containerID="49730f2775cb8087334f1b59907ecb0576364b3b1c0de69f0ed9302b19db2e7c" exitCode=0 Jul 13 15:38:45.563401 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:45.563075 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682982229312" event={"ID":"e10b68c1-bf89-4fc6-b3f2-484c76c569cf","Type":"ContainerDied","Data":"49730f2775cb8087334f1b59907ecb0576364b3b1c0de69f0ed9302b19db2e7c"} Jul 13 15:38:45.582092 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:45.582060 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682983773936\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" pod="openshift-ingress/test-curl-7715-140682983773936" Jul 13 15:38:45.679187 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:45.679149 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0777bc7-6c92-4bc6-a898-40e0fb402f5b" path="/var/lib/kubelet/pods/f0777bc7-6c92-4bc6-a898-40e0fb402f5b/volumes" Jul 13 15:38:46.716825 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.716798 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:46.744476 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.744443 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682982229312_e10b68c1-bf89-4fc6-b3f2-484c76c569cf/test-curl-7715-140682982229312/0.log" Jul 13 15:38:46.769907 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.769875 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"] Jul 13 15:38:46.773930 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.773900 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"] Jul 13 15:38:46.776872 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.776853 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h99h2\" (UniqueName: \"kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2\") pod \"e10b68c1-bf89-4fc6-b3f2-484c76c569cf\" (UID: \"e10b68c1-bf89-4fc6-b3f2-484c76c569cf\") " Jul 13 15:38:46.779051 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.779028 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2" (OuterVolumeSpecName: "kube-api-access-h99h2") pod "e10b68c1-bf89-4fc6-b3f2-484c76c569cf" (UID: "e10b68c1-bf89-4fc6-b3f2-484c76c569cf"). InnerVolumeSpecName "kube-api-access-h99h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:38:46.877944 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:46.877868 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h99h2\" (UniqueName: \"kubernetes.io/projected/e10b68c1-bf89-4fc6-b3f2-484c76c569cf-kube-api-access-h99h2\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:47.576461 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:47.576431 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:47.576461 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:47.576462 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ff5af82dcb42d7a95f8c8cbbc50814e567090d16896436d2d80d0c258feb85" Jul 13 15:38:47.586536 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:47.586512 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682982229312\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:47.678206 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:38:47.678170 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7715-140682982229312\" is forbidden: User \"system:node:ip-10-0-134-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-241.ec2.internal' and this object" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" pod="openshift-ingress/test-curl-7715-140682982229312" Jul 13 15:38:47.680078 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:47.680057 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" path="/var/lib/kubelet/pods/e10b68c1-bf89-4fc6-b3f2-484c76c569cf/volumes" Jul 13 15:38:50.454030 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.453993 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h"] Jul 13 15:38:50.454441 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.454428 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" containerName="test-curl-7715-140682982229312" Jul 13 15:38:50.454507 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.454443 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" containerName="test-curl-7715-140682982229312" Jul 13 15:38:50.454546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.454523 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10b68c1-bf89-4fc6-b3f2-484c76c569cf" containerName="test-curl-7715-140682982229312" Jul 13 15:38:50.457744 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.457724 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:50.460313 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.460292 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\"" Jul 13 15:38:50.466065 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.466043 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h"] Jul 13 15:38:50.508874 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.508849 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-88b181-xg66h\" (UID: \"dd9f6ae0-90a8-4146-9276-7f57750b42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:50.609769 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.609737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-88b181-xg66h\" (UID: \"dd9f6ae0-90a8-4146-9276-7f57750b42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:50.610398 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.610379 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-88b181-xg66h\" (UID: \"dd9f6ae0-90a8-4146-9276-7f57750b42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:50.770408 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.770377 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:50.910804 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:50.910762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h"] Jul 13 15:38:50.912166 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:38:50.912142 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9f6ae0_90a8_4146_9276_7f57750b42ac.slice/crio-e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed WatchSource:0}: Error finding container e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed: Status 404 returned error can't find the container with id e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed Jul 13 15:38:51.046393 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:51.046319 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:38:51.149279 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:51.149252 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:38:51.601689 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:51.601648 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" event={"ID":"dd9f6ae0-90a8-4146-9276-7f57750b42ac","Type":"ContainerStarted","Data":"d798a742033ac0381556ae7890d16c65f2823186e975388469ee46faa9f0ec4d"} Jul 13 15:38:51.602185 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:51.601746 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" event={"ID":"dd9f6ae0-90a8-4146-9276-7f57750b42ac","Type":"ContainerStarted","Data":"e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed"} Jul 13 15:38:51.621065 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:51.621013 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" podStartSLOduration=1.6209965149999999 podStartE2EDuration="1.620996515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:38:51.618664395 +0000 UTC m=+2794.509339498" watchObservedRunningTime="2026-07-13 15:38:51.620996515 +0000 UTC m=+2794.511671620" Jul 13 15:38:52.609445 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:52.609409 2554 generic.go:356] "Generic (PLEG): container finished" podID="dd9f6ae0-90a8-4146-9276-7f57750b42ac" containerID="d798a742033ac0381556ae7890d16c65f2823186e975388469ee46faa9f0ec4d" exitCode=0 Jul 13 15:38:52.609837 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:52.609485 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" event={"ID":"dd9f6ae0-90a8-4146-9276-7f57750b42ac","Type":"ContainerDied","Data":"d798a742033ac0381556ae7890d16c65f2823186e975388469ee46faa9f0ec4d"} Jul 13 15:38:53.757546 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:53.757520 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:53.839742 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:53.839709 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca\") pod \"dd9f6ae0-90a8-4146-9276-7f57750b42ac\" (UID: \"dd9f6ae0-90a8-4146-9276-7f57750b42ac\") " Jul 13 15:38:53.840110 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:53.840088 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd9f6ae0-90a8-4146-9276-7f57750b42ac" (UID: "dd9f6ae0-90a8-4146-9276-7f57750b42ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:38:53.941396 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:53.941319 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f6ae0-90a8-4146-9276-7f57750b42ac-maas-api-service-ca\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:38:54.623104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:54.623076 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" Jul 13 15:38:54.623104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:54.623098 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-88b181-xg66h" event={"ID":"dd9f6ae0-90a8-4146-9276-7f57750b42ac","Type":"ContainerDied","Data":"e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed"} Jul 13 15:38:54.623314 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:38:54.623141 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4a0ea4831a5bbe8ee164bc443ec8f491a0b1f9516daa784ae8bd9111410b5ed" Jul 13 15:39:00.900193 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:00.900162 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:05.550802 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:05.550749 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:11.565416 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:11.565360 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:11.627089 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:11.627060 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:16.360621 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:16.360588 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:16.601887 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:16.601850 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:17.248156 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:17.248124 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:17.345842 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:17.345805 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:27.218423 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:27.218391 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:31.851803 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:31.851750 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:33.075667 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:33.075636 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:33.176379 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:33.176346 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:37.657527 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:37.657494 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:37.720984 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:37.720949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:42.546469 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:42.546433 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:39:42.546878 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:42.546678 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" containerID="cri-o://9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8" gracePeriod=30 Jul 13 15:39:42.568272 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:42.568176 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:42.836494 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:42.836415 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:39:47.147740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.147697 2554 patch_prober.go:28] interesting pod/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 15:39:47.148148 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.147776 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 15:39:47.813319 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.813257 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:39:47.932584 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.932543 2554 generic.go:356] "Generic (PLEG): container finished" podID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerID="9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8" exitCode=0 Jul 13 15:39:47.932776 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.932591 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" Jul 13 15:39:47.932776 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.932623 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerDied","Data":"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8"} Jul 13 15:39:47.932776 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.932671 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs" event={"ID":"b214ba35-5b22-4dd8-b1ef-90f60b2d8979","Type":"ContainerDied","Data":"a9c57e2a08340eb7baa5e60bee184b277c8b5c003bad9c4d802d6cbddddf8936"} Jul 13 15:39:47.932776 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.932694 2554 scope.go:117] "RemoveContainer" containerID="9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8" Jul 13 15:39:47.941821 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.941779 2554 scope.go:117] "RemoveContainer" containerID="a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796" Jul 13 15:39:47.949618 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.949601 2554 scope.go:117] "RemoveContainer" containerID="9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8" Jul 13 15:39:47.949889 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:39:47.949868 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8\": container with ID starting with 9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8 not found: ID does not exist" containerID="9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8" Jul 13 15:39:47.949955 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.949899 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8"} err="failed to get container status \"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8\": rpc error: code = NotFound desc = could not find container \"9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8\": container with ID starting with 9065d6d643bb236745cf0357e9b27cce8caef41fc58a5f7658c8811f500f1dc8 not found: ID does not exist" Jul 13 15:39:47.949955 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.949915 2554 scope.go:117] "RemoveContainer" containerID="a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796" Jul 13 15:39:47.950161 ip-10-0-134-241 kubenswrapper[2554]: E0713 15:39:47.950143 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796\": container with ID starting with a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796 not found: ID does not exist" containerID="a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796" Jul 13 15:39:47.950213 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.950167 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796"} err="failed to get container status \"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796\": rpc error: code = NotFound desc = could not find container \"a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796\": container with ID starting with a52989b84093be3287586f97651615724a60aaf89bd5b4b181ce0227b4c20796 not found: ID does not exist" Jul 13 15:39:47.968088 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968068 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968172 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968098 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968172 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968127 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968174 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968211 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968267 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968262 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968419 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968279 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968419 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968303 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968419 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968325 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh5cf\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf\") pod \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\" (UID: \"b214ba35-5b22-4dd8-b1ef-90f60b2d8979\") " Jul 13 15:39:47.968419 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968336 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:39:47.968419 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968348 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:39:47.968679 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968608 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-credential-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:47.968679 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968629 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-certs\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:47.968679 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968652 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:39:47.968864 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.968830 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:39:47.969142 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.969118 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data" (OuterVolumeSpecName: "istio-data") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:39:47.970456 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.970429 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token" (OuterVolumeSpecName: "istio-token") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:39:47.970524 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.970497 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf" (OuterVolumeSpecName: "kube-api-access-bh5cf") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "kube-api-access-bh5cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:39:47.970674 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.970656 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:39:47.970738 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:47.970675 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b214ba35-5b22-4dd8-b1ef-90f60b2d8979" (UID: "b214ba35-5b22-4dd8-b1ef-90f60b2d8979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070298 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-podinfo\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070327 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-envoy\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070338 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-token\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070349 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-workload-socket\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070360 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istio-data\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070381 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070372 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5cf\" (UniqueName: \"kubernetes.io/projected/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-kube-api-access-bh5cf\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.070668 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.070386 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b214ba35-5b22-4dd8-b1ef-90f60b2d8979-istiod-ca-cert\") on node \"ip-10-0-134-241.ec2.internal\" DevicePath \"\"" Jul 13 15:39:48.256855 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.256825 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:39:48.260949 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.260923 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-450dce-openshift-default-86ddc5859-lqmgs"] Jul 13 15:39:48.857074 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.857040 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bj4g4_5e828e4f-345b-4c09-aa57-f5fdb792d869/manager/0.log" Jul 13 15:39:48.984458 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:48.984430 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9c8dd9b7-dkrtr_bc31e8d0-a137-4fac-9233-99094be96b31/manager/0.log" Jul 13 15:39:49.107885 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:49.107810 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fx4tk_8798a704-483e-4d73-b938-df6e2bc2ea38/server/0.log" Jul 13 15:39:49.226556 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:49.226524 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lfjl4_1b92b726-0be8-4ef0-a5a0-951fd9602788/manager/0.log" Jul 13 15:39:49.353479 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:49.353451 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-h5xt2_3c77deb1-ddf1-465e-b47b-8fd41a709c4e/copy-manifests/0.log" Jul 13 15:39:49.369343 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:49.369279 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-h5xt2_3c77deb1-ddf1-465e-b47b-8fd41a709c4e/manager/0.log" Jul 13 15:39:49.679232 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:49.679150 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" path="/var/lib/kubelet/pods/b214ba35-5b22-4dd8-b1ef-90f60b2d8979/volumes" Jul 13 15:39:50.046660 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.046630 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86d5b84cf-thz6z_25bc57b3-2783-45a4-ad0c-48c314b81b1a/maas-api/0.log" Jul 13 15:39:50.172211 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.172187 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732595-gqffc_2bf4a955-b627-4eeb-a192-ea834ed68f10/cleanup/0.log" Jul 13 15:39:50.297847 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.297751 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732610-c4mst_3030602b-edef-4f56-9f73-d50fd834817d/cleanup/0.log" Jul 13 15:39:50.656434 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.656361 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-4b866e-62pwk_c023008b-29be-4295-9b3d-aa272edcb870/revoke-keys/0.log" Jul 13 15:39:50.774911 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.774879 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-d3da1a-5cszt_7d586b70-1493-4485-8b51-0e479ecfe1a1/revoke-keys/0.log" Jul 13 15:39:50.890850 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:50.890823 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-f99515a5-qkgqb_bba4d722-9097-4727-8cd7-01f2409b5f5c/revoke-keys/0.log" Jul 13 15:39:51.011479 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:51.011453 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-15d63b-6vvkw_a3695726-9bd0-4aa1-a08b-cf0e3fb036d7/revoke-keys/0.log" Jul 13 15:39:51.130360 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:51.130331 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-f68651-xmm56_96df2bcf-cbb4-4239-b7e4-fa80490e50b0/revoke-keys/0.log" Jul 13 15:39:51.500740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:51.500664 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-fb1236df-2n5tc_cc41157c-5c08-464f-90fe-aed10e28f279/revoke-keys/0.log" Jul 13 15:39:51.619853 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:51.619824 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-88b181-xg66h_dd9f6ae0-90a8-4146-9276-7f57750b42ac/revoke-keys/0.log" Jul 13 15:39:51.860532 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:51.860502 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sdm6m_8f176e3d-2acf-4858-9e3e-709cb8d68734/postgres/0.log" Jul 13 15:39:53.405836 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:53.405807 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7mbwr_89acfeb7-902e-4d35-a33d-1d0742674e73/manager/0.log" Jul 13 15:39:53.538215 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:53.538186 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-77s8v_27bffcd5-5444-4e76-bc2a-d574dfa3768c/manager/0.log" Jul 13 15:39:53.812427 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:53.812396 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lvzg2_92c831c8-76fe-4e79-9504-e00004db8d93/registry-server/0.log" Jul 13 15:39:53.946755 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:53.946729 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xg5cx_b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1/manager/0.log" Jul 13 15:39:54.980492 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:54.980463 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-qktng_b2630a39-f876-4377-b863-d43701d44e92/kube-auth-proxy/0.log" Jul 13 15:39:55.109298 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:55.109269 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/1.log" Jul 13 15:39:55.242041 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:55.241969 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-bpgmt_aea2ef8e-4abe-4918-94df-6e7d3e23f492/payload-pre-processing/0.log" Jul 13 15:39:55.384577 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:55.384550 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-v5f85_622f4a8e-9934-4543-90b4-bec1dd3ce4a0/payload-processing/0.log" Jul 13 15:39:55.507009 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:55.506935 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66644b4cfd-4t6dc_254d637f-eb4a-4b2e-ac6b-9ac05474709b/router/0.log" Jul 13 15:39:56.000410 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:56.000383 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jphqm_9430f0cd-ac95-422b-8863-04028ea11020/main/0.log" Jul 13 15:39:56.133256 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:56.133224 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k8cnw_5653cfd3-94ff-41ef-9818-d07f2e721451/main/0.log" Jul 13 15:39:58.599464 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:39:58.599424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:40:00.516007 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.515971 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdpg8/must-gather-kq787"] Jul 13 15:40:00.516404 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516391 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.516454 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516405 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.516454 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516421 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9f6ae0-90a8-4146-9276-7f57750b42ac" containerName="revoke-keys" Jul 13 15:40:00.516454 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516428 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9f6ae0-90a8-4146-9276-7f57750b42ac" containerName="revoke-keys" Jul 13 15:40:00.516551 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516497 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.516551 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516505 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.516551 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516514 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9f6ae0-90a8-4146-9276-7f57750b42ac" containerName="revoke-keys" Jul 13 15:40:00.516639 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516602 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.516639 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.516609 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b214ba35-5b22-4dd8-b1ef-90f60b2d8979" containerName="istio-proxy" Jul 13 15:40:00.520017 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.520000 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.523176 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.523158 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vdpg8\"/\"default-dockercfg-sdvkg\"" Jul 13 15:40:00.524098 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.524081 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdpg8\"/\"kube-root-ca.crt\"" Jul 13 15:40:00.524198 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.524081 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdpg8\"/\"openshift-service-ca.crt\"" Jul 13 15:40:00.530807 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.530771 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/must-gather-kq787"] Jul 13 15:40:00.537125 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.537097 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:40:00.636433 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.636402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" limit=1024 Jul 13 15:40:00.684980 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.684951 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15ae563d-3391-459d-809b-b65835ad4c8e-must-gather-output\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.685147 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.685056 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgwf\" (UniqueName: \"kubernetes.io/projected/15ae563d-3391-459d-809b-b65835ad4c8e-kube-api-access-tbgwf\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.786127 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.786043 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgwf\" (UniqueName: \"kubernetes.io/projected/15ae563d-3391-459d-809b-b65835ad4c8e-kube-api-access-tbgwf\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.786127 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.786096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15ae563d-3391-459d-809b-b65835ad4c8e-must-gather-output\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.786473 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.786454 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15ae563d-3391-459d-809b-b65835ad4c8e-must-gather-output\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.794573 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.794549 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgwf\" (UniqueName: \"kubernetes.io/projected/15ae563d-3391-459d-809b-b65835ad4c8e-kube-api-access-tbgwf\") pod \"must-gather-kq787\" (UID: \"15ae563d-3391-459d-809b-b65835ad4c8e\") " pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.830648 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.830620 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/must-gather-kq787" Jul 13 15:40:00.971762 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:00.971740 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/must-gather-kq787"] Jul 13 15:40:00.974447 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:40:00.974420 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ae563d_3391_459d_809b_b65835ad4c8e.slice/crio-623289130bed23013205f28012e7622d167eb120efda9fccfa6d621dff81e737 WatchSource:0}: Error finding container 623289130bed23013205f28012e7622d167eb120efda9fccfa6d621dff81e737: Status 404 returned error can't find the container with id 623289130bed23013205f28012e7622d167eb120efda9fccfa6d621dff81e737 Jul 13 15:40:01.007586 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:01.007561 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/must-gather-kq787" event={"ID":"15ae563d-3391-459d-809b-b65835ad4c8e","Type":"ContainerStarted","Data":"623289130bed23013205f28012e7622d167eb120efda9fccfa6d621dff81e737"} Jul 13 15:40:02.017251 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:02.017217 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/must-gather-kq787" event={"ID":"15ae563d-3391-459d-809b-b65835ad4c8e","Type":"ContainerStarted","Data":"c7d7bb7304ad4c4cf06734d0f4e52d92129e5d7641aee6b8cd90000a4402ee99"} Jul 13 15:40:02.017891 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:02.017838 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/must-gather-kq787" event={"ID":"15ae563d-3391-459d-809b-b65835ad4c8e","Type":"ContainerStarted","Data":"3f69c8c406676e125e52ffac51f4c862bd64a51e369f199b2e54afb7d72b01dd"} Jul 13 15:40:02.037976 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:02.037909 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdpg8/must-gather-kq787" podStartSLOduration=1.401569717 podStartE2EDuration="2.037887877s" totalImagesPullingTime="636.31816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:40:02.036050119 +0000 UTC m=+2864.926725243" watchObservedRunningTime="2026-07-13 15:40:02.037887877 +0000 UTC m=+2864.928562981" Jul 13 15:40:03.311183 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:03.311157 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6mfkf_b09ff867-1f8f-40fc-883f-7afd37b3b88b/global-pull-secret-syncer/0.log" Jul 13 15:40:03.464572 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:03.464531 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wf4g_02703252-d08b-45ff-9716-08cbe91b83e1/konnectivity-agent/0.log" Jul 13 15:40:03.617153 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:03.617069 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-241.ec2.internal_98260bfd566f14ae3c27b8cadea6e987/haproxy/0.log" Jul 13 15:40:08.610482 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:08.610446 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7mbwr_89acfeb7-902e-4d35-a33d-1d0742674e73/manager/0.log" Jul 13 15:40:08.665873 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:08.665831 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-77s8v_27bffcd5-5444-4e76-bc2a-d574dfa3768c/manager/0.log" Jul 13 15:40:08.749876 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:08.749843 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lvzg2_92c831c8-76fe-4e79-9504-e00004db8d93/registry-server/0.log" Jul 13 15:40:08.899418 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:08.899338 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xg5cx_b5f092ed-5c3f-4ed2-90eb-f397ad18f7a1/manager/0.log" Jul 13 15:40:10.819567 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:10.819530 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-pb9f5_190cd310-f8ad-4148-85d7-cd456b1d1552/cluster-monitoring-operator/0.log" Jul 13 15:40:11.091317 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:11.091286 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndcgp_06c3f473-a1b4-45e3-87ee-e1c87458f4cf/node-exporter/0.log" Jul 13 15:40:11.116104 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:11.116081 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndcgp_06c3f473-a1b4-45e3-87ee-e1c87458f4cf/kube-rbac-proxy/0.log" Jul 13 15:40:11.139978 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:11.139948 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndcgp_06c3f473-a1b4-45e3-87ee-e1c87458f4cf/init-textfile/0.log" Jul 13 15:40:11.775212 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:11.775177 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-wjjb9_e81b452e-5c6f-4220-b702-d7f0801d8ab3/prometheus-operator-admission-webhook/0.log" Jul 13 15:40:13.362279 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:13.362233 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-8km9s_e632f3ac-8807-4f49-b083-2b41f69e6890/networking-console-plugin/0.log" Jul 13 15:40:13.939046 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:13.939011 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/1.log" Jul 13 15:40:13.946331 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:13.946301 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mbl9k_c8b6e5d5-b197-4dfc-a471-496cbd3d24ee/console-operator/2.log" Jul 13 15:40:14.029262 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.029225 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr"] Jul 13 15:40:14.035993 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.035965 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.041865 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.041832 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcth\" (UniqueName: \"kubernetes.io/projected/ed778151-6e00-44c4-ad8d-2477b1c6afc5-kube-api-access-hdcth\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.042006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.041926 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-lib-modules\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.042006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.041967 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-proc\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.042006 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.042001 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-sys\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.042773 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.042066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-podres\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.044961 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.044942 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr"] Jul 13 15:40:14.143833 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143779 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-lib-modules\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-proc\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-sys\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143908 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-podres\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143930 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcth\" (UniqueName: \"kubernetes.io/projected/ed778151-6e00-44c4-ad8d-2477b1c6afc5-kube-api-access-hdcth\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143952 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-lib-modules\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143929 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-proc\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144024 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.143993 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-sys\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.144277 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.144062 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed778151-6e00-44c4-ad8d-2477b1c6afc5-podres\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.154281 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.154256 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcth\" (UniqueName: \"kubernetes.io/projected/ed778151-6e00-44c4-ad8d-2477b1c6afc5-kube-api-access-hdcth\") pod \"perf-node-gather-daemonset-8ckzr\" (UID: \"ed778151-6e00-44c4-ad8d-2477b1c6afc5\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.354288 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.354244 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:14.455992 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.455967 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b84bd7d78-jgzmj_295dbd78-78e9-4ee6-8aeb-4d270d341ae9/console/0.log" Jul 13 15:40:14.503705 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.503681 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-6nd4v_6c4e880c-1abc-498e-9179-3aa3fe532d2b/download-server/0.log" Jul 13 15:40:14.526241 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:14.526214 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr"] Jul 13 15:40:14.527732 ip-10-0-134-241 kubenswrapper[2554]: W0713 15:40:14.527689 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poded778151_6e00_44c4_ad8d_2477b1c6afc5.slice/crio-4793b31ef53d486aab4c68d77454f37d84a3faa781ce9945005cb6b942fb84b1 WatchSource:0}: Error finding container 4793b31ef53d486aab4c68d77454f37d84a3faa781ce9945005cb6b942fb84b1: Status 404 returned error can't find the container with id 4793b31ef53d486aab4c68d77454f37d84a3faa781ce9945005cb6b942fb84b1 Jul 13 15:40:15.129663 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:15.129639 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-6jjts_eca0983c-d818-4ffd-a300-f985896230d5/volume-data-source-validator/0.log" Jul 13 15:40:15.140047 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:15.140013 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" event={"ID":"ed778151-6e00-44c4-ad8d-2477b1c6afc5","Type":"ContainerStarted","Data":"e7fd755df9290027c0b57f25ebd0c7fca9349eda567c97e554c0dcaf8ea818b5"} Jul 13 15:40:15.140209 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:15.140067 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" event={"ID":"ed778151-6e00-44c4-ad8d-2477b1c6afc5","Type":"ContainerStarted","Data":"4793b31ef53d486aab4c68d77454f37d84a3faa781ce9945005cb6b942fb84b1"} Jul 13 15:40:15.140209 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:15.140133 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:15.161126 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:15.161065 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" podStartSLOduration=1.161044093 podStartE2EDuration="1.161044093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 15:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:40:15.157935054 +0000 UTC m=+2878.048610157" watchObservedRunningTime="2026-07-13 15:40:15.161044093 +0000 UTC m=+2878.051719196" Jul 13 15:40:16.214046 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:16.213970 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c9s7d_d1f2e019-aaa0-4042-99df-69b9fd53c892/dns/0.log" Jul 13 15:40:16.237908 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:16.237873 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c9s7d_d1f2e019-aaa0-4042-99df-69b9fd53c892/kube-rbac-proxy/0.log" Jul 13 15:40:16.341617 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:16.341593 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m2k8m_c601ac53-21fc-4809-80e6-b8cdc296e7e7/dns-node-resolver/0.log" Jul 13 15:40:16.962562 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:16.962531 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bvxdz_56e9f752-fd9a-4f05-8c86-fb30760212c3/node-ca/0.log" Jul 13 15:40:18.262993 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.262967 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-qktng_b2630a39-f876-4377-b863-d43701d44e92/kube-auth-proxy/0.log" Jul 13 15:40:18.295987 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.295960 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/0.log" Jul 13 15:40:18.332025 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.331953 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/1.log" Jul 13 15:40:18.368588 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.368559 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-bpgmt_aea2ef8e-4abe-4918-94df-6e7d3e23f492/payload-pre-processing/0.log" Jul 13 15:40:18.406624 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.406596 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-v5f85_622f4a8e-9934-4543-90b4-bec1dd3ce4a0/payload-processing/0.log" Jul 13 15:40:18.435658 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:18.435630 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66644b4cfd-4t6dc_254d637f-eb4a-4b2e-ac6b-9ac05474709b/router/0.log" Jul 13 15:40:19.234156 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.234125 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zjf24_7d2dec88-ccda-4b33-85f7-870aece5497b/serve-healthcheck-canary/0.log" Jul 13 15:40:19.732243 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.732206 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-j4cn7_35ca4899-57dd-4122-a6aa-031e069d9f82/insights-operator/0.log" Jul 13 15:40:19.732772 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.732755 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-j4cn7_35ca4899-57dd-4122-a6aa-031e069d9f82/insights-operator/1.log" Jul 13 15:40:19.756609 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.756582 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9svjz_0ab3b13b-61b0-4ab3-a738-6173ab8b4749/kube-rbac-proxy/0.log" Jul 13 15:40:19.780657 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.780634 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9svjz_0ab3b13b-61b0-4ab3-a738-6173ab8b4749/exporter/0.log" Jul 13 15:40:19.805369 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:19.805344 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9svjz_0ab3b13b-61b0-4ab3-a738-6173ab8b4749/extractor/0.log" Jul 13 15:40:21.155602 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:21.155576 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-8ckzr" Jul 13 15:40:22.261554 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.261528 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bj4g4_5e828e4f-345b-4c09-aa57-f5fdb792d869/manager/0.log" Jul 13 15:40:22.400864 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.400837 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9c8dd9b7-dkrtr_bc31e8d0-a137-4fac-9233-99094be96b31/manager/0.log" Jul 13 15:40:22.434703 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.434668 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fx4tk_8798a704-483e-4d73-b938-df6e2bc2ea38/server/0.log" Jul 13 15:40:22.541598 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.541507 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lfjl4_1b92b726-0be8-4ef0-a5a0-951fd9602788/manager/0.log" Jul 13 15:40:22.633837 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.633803 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-h5xt2_3c77deb1-ddf1-465e-b47b-8fd41a709c4e/manager/0.log" Jul 13 15:40:22.654412 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:22.654383 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-h5xt2_3c77deb1-ddf1-465e-b47b-8fd41a709c4e/copy-manifests/0.log" Jul 13 15:40:23.971740 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:23.971716 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-qktng_b2630a39-f876-4377-b863-d43701d44e92/kube-auth-proxy/0.log" Jul 13 15:40:23.997586 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:23.997422 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/0.log" Jul 13 15:40:24.046963 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:24.046917 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/1.log" Jul 13 15:40:24.070967 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:24.070926 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-bpgmt_aea2ef8e-4abe-4918-94df-6e7d3e23f492/payload-pre-processing/0.log" Jul 13 15:40:24.097960 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:24.097932 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-v5f85_622f4a8e-9934-4543-90b4-bec1dd3ce4a0/payload-processing/0.log" Jul 13 15:40:24.120500 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:24.120472 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66644b4cfd-4t6dc_254d637f-eb4a-4b2e-ac6b-9ac05474709b/router/0.log" Jul 13 15:40:25.511723 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.511680 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-qktng_b2630a39-f876-4377-b863-d43701d44e92/kube-auth-proxy/0.log" Jul 13 15:40:25.540741 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.540698 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/0.log" Jul 13 15:40:25.582150 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.582119 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wggx2_bece848d-6294-46cc-b9da-d2e5f1436678/istio-proxy/1.log" Jul 13 15:40:25.603759 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.603731 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-bpgmt_aea2ef8e-4abe-4918-94df-6e7d3e23f492/payload-pre-processing/0.log" Jul 13 15:40:25.628245 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.628204 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-v5f85_622f4a8e-9934-4543-90b4-bec1dd3ce4a0/payload-processing/0.log" Jul 13 15:40:25.645780 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:25.645753 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66644b4cfd-4t6dc_254d637f-eb4a-4b2e-ac6b-9ac05474709b/router/0.log" Jul 13 15:40:29.353388 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:29.353360 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-w6k66_d746f33c-9114-40c8-b256-1bb5d0d36485/migrator/0.log" Jul 13 15:40:29.380194 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:29.380172 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-w6k66_d746f33c-9114-40c8-b256-1bb5d0d36485/graceful-termination/0.log" Jul 13 15:40:29.748048 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:29.747968 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-q6bnd_bc3a4046-3553-4864-a26c-59f35f1f0da9/kube-storage-version-migrator-operator/1.log" Jul 13 15:40:29.749505 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:29.749478 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-q6bnd_bc3a4046-3553-4864-a26c-59f35f1f0da9/kube-storage-version-migrator-operator/0.log" Jul 13 15:40:31.076939 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.076910 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/kube-multus-additional-cni-plugins/0.log" Jul 13 15:40:31.103044 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.103018 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/egress-router-binary-copy/0.log" Jul 13 15:40:31.150830 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.150752 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/cni-plugins/0.log" Jul 13 15:40:31.209412 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.209386 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/bond-cni-plugin/0.log" Jul 13 15:40:31.242824 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.242778 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/routeoverride-cni/0.log" Jul 13 15:40:31.276119 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.276086 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/whereabouts-cni-bincopy/0.log" Jul 13 15:40:31.305296 ip-10-0-134-241 kubenswrapper[2554]: I0713 15:40:31.305269 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m7rh6_160c96bd-e5f1-4bc8-9788-14dee3ac8af6/whereabouts-cni/0.log"