Jul 20 02:45:05.004577 ip-10-0-134-38 systemd[1]: Starting Kubernetes Kubelet... Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.802822 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.808980 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.808992 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.808997 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809001 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809005 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809008 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809011 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.843065 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809014 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809018 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809020 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809023 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809026 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809029 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809031 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809035 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809038 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809041 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809044 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809046 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809049 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809052 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809055 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809057 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809060 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809063 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809066 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809069 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.853952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809071 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809074 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809077 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809081 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809084 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809087 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809090 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809093 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809095 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809098 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809101 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809104 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809108 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809111 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809113 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809116 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809119 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809122 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809125 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809128 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.855558 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809131 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809134 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809137 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809139 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809142 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809146 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809151 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809154 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809157 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809160 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809163 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809165 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809168 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809171 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809174 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809177 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809180 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809183 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809186 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809188 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.856664 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809191 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809196 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809200 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809203 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809206 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809209 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809212 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809214 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809217 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809220 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809222 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809225 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809228 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809231 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809234 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809236 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809239 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809241 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809247 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.968007 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809250 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.881586 ip-10-0-134-38 systemd[1]: Started Kubernetes Kubelet. Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809254 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809257 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809260 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809263 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809266 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809269 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809272 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809275 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809278 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809280 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809284 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.809287 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810077 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810085 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810089 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810093 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810095 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810098 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.969620 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810102 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810105 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810107 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810110 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810113 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810116 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810119 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810122 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810125 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810128 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810130 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810133 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810136 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810139 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810142 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810144 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810147 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810150 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810153 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810155 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.971187 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810158 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810161 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810164 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810167 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810171 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810173 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810176 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810179 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810181 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810185 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810187 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810190 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810193 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810197 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810202 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810206 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810209 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810212 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810215 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.972300 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810218 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810221 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810225 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810228 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810231 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810234 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810237 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810240 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810242 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810245 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810248 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810251 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810254 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810256 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810259 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810263 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810267 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810270 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810273 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810276 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.973882 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810279 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810282 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810286 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810289 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810293 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810296 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810302 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810305 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810308 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810311 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810313 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810316 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810318 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810322 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810325 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810328 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810332 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810335 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810338 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810341 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.976492 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810344 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810347 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810349 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810352 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810355 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810358 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810360 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810379 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810383 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810386 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810389 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810391 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810394 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810397 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810794 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810799 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810802 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810805 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810809 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.977374 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810812 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810815 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810818 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810820 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810823 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810826 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810829 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810832 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810834 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810837 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810840 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810843 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810845 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810848 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810851 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810854 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810856 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810860 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810862 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810865 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.978147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810868 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810871 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810874 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810877 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810879 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810882 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810884 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810887 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810890 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810893 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810896 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810901 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810905 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810908 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810911 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810913 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810916 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810919 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810922 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810925 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.978952 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810928 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810930 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810933 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810936 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810939 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810941 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810944 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810947 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810949 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810952 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810955 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810957 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810960 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810963 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810966 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810968 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810971 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810974 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810976 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810979 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.979772 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810982 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810986 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810989 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810994 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.810998 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811001 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811004 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811008 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811012 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811020 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811023 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811026 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811029 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811032 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811035 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811037 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811040 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811043 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811046 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.980596 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811048 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811051 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811054 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811056 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811059 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811061 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811064 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811067 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811126 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811170 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811174 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811177 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811180 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811183 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.811186 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811431 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811462 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811475 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811483 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811491 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811496 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 02:45:06.981407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811503 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811511 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811516 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811527 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811533 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811540 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811545 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811550 2565 flags.go:64] FLAG: --cgroup-root="" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811555 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811560 2565 flags.go:64] FLAG: --client-ca-file="" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811566 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811571 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811589 2565 flags.go:64] FLAG: --cluster-domain="" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811594 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811600 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811606 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811612 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811619 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811624 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811630 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811641 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811646 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811652 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811658 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 02:45:06.982257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811664 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811669 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811687 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811694 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811704 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811709 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811715 2565 flags.go:64] FLAG: --enable-server="true" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811720 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811728 2565 flags.go:64] FLAG: --event-burst="100" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811733 2565 flags.go:64] FLAG: --event-qps="50" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811738 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811744 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811749 2565 flags.go:64] FLAG: --eviction-hard="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811760 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811765 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811771 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811776 2565 flags.go:64] FLAG: --eviction-soft="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811781 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811787 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811792 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811797 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811807 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811812 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811818 2565 flags.go:64] FLAG: --feature-gates="" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811825 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 02:45:06.983213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811830 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811836 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811842 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811847 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811853 2565 flags.go:64] FLAG: --help="false" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811865 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-134-38.ec2.internal" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811871 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811876 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811882 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811889 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811896 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811903 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811908 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811919 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811925 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811930 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811936 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811941 2565 flags.go:64] FLAG: --kube-reserved="" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811947 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811952 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811958 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811964 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811974 2565 flags.go:64] FLAG: --lock-file="" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811979 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 02:45:06.984234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811985 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.811991 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812001 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812006 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812012 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812017 2565 flags.go:64] FLAG: --logging-format="text" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812027 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812033 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812038 2565 flags.go:64] FLAG: --manifest-url="" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812044 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812052 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812058 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812066 2565 flags.go:64] FLAG: --max-pods="110" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812072 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812085 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812091 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812096 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812102 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812107 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812113 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812127 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812133 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812144 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 02:45:06.985217 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812149 2565 flags.go:64] FLAG: --pod-cidr="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812155 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812239 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812355 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812369 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812390 2565 flags.go:64] FLAG: --port="10250" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812394 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812398 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0857d2432b1cb8b" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812402 2565 flags.go:64] FLAG: --qos-reserved="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812405 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812409 2565 flags.go:64] FLAG: --register-node="true" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812412 2565 flags.go:64] FLAG: --register-with-taints="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812417 2565 flags.go:64] FLAG: --registry-burst="10" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812420 2565 flags.go:64] FLAG: --registry-qps="5" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812423 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812426 2565 flags.go:64] FLAG: --reserved-memory="" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812430 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812434 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812437 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812440 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812444 2565 flags.go:64] FLAG: --runonce="false" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812447 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812453 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812456 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812461 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 02:45:06.986158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812464 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812468 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812471 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812474 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812477 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812482 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812485 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812488 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812491 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812494 2565 flags.go:64] FLAG: --system-cgroups="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812497 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812507 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812511 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812514 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812519 2565 flags.go:64] FLAG: --tls-min-version="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812522 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812526 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812529 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812532 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812535 2565 flags.go:64] FLAG: --v="2" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812540 2565 flags.go:64] FLAG: --version="false" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812544 2565 flags.go:64] FLAG: --vmodule="" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812548 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.812552 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812685 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.987237 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812689 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812694 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812697 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812701 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812707 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812710 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812715 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812718 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812721 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812724 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812727 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812729 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812733 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812735 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812738 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812741 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812744 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812748 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812752 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.988247 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812755 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812758 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812761 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812764 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812766 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812769 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812772 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812775 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812777 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812780 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812783 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812786 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812789 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812792 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812795 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812798 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812801 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812805 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812808 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.989046 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812812 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812815 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812818 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812821 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812823 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812827 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812829 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812834 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812838 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812841 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812844 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812847 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812851 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812853 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812856 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812859 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812863 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812867 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812870 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.989833 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812873 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812877 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812880 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812883 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812886 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812889 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812891 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812894 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812897 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812900 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812903 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812907 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812910 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812914 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812917 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812919 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812922 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812925 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812928 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812931 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.990606 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812934 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812936 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812939 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812942 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812945 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812948 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812950 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812953 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812956 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812960 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812963 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812965 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812969 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812972 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812975 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812977 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812980 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812983 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812986 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812988 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.991418 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.812991 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.813590 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.821023 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.821041 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.822242 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.822251 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822312 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822317 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822321 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822323 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822326 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822329 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822333 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822337 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822342 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.992212 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822346 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822349 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822353 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822356 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822359 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822362 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822381 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822384 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822388 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822390 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822393 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822396 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822399 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822402 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822406 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822410 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822413 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822416 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822419 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.992848 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822423 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822428 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822431 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822435 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822438 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822441 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822444 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822447 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822450 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822453 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822455 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822458 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822461 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822464 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822467 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822469 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822472 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822475 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822478 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822481 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.993629 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822484 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822486 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822489 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822492 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822495 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822497 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822500 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822503 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822507 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822509 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822512 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822518 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822520 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822525 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822528 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822531 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822534 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822537 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822540 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822543 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:06.994458 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822546 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822549 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822551 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822554 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822557 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822560 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822563 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822566 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822569 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822573 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822577 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822580 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822583 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822586 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822588 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822591 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822594 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822597 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822599 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.995259 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822603 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822606 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822609 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822611 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822616 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822618 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822623 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822626 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822629 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822632 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822634 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822637 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.822643 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822762 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822768 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:06.996100 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822771 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822774 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822777 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822780 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822783 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822785 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822788 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822792 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822796 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822800 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822803 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822806 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822809 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822812 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822815 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822818 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822821 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822825 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822828 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:06.996734 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822831 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822833 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822836 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822841 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822846 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822849 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822851 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822855 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822857 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822861 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822864 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822866 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822869 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822872 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822875 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822878 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822881 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822884 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822886 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822889 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:06.997516 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822892 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822894 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822897 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822900 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822902 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822906 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822910 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822913 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822916 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822919 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822922 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822924 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822927 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822930 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822933 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822937 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822941 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822944 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822947 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:06.998332 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822950 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822953 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822956 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822959 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822961 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822964 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822967 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822969 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822972 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822975 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822977 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822980 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822983 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822985 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822988 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822991 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822994 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822997 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.822999 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:06.999129 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823003 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823007 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823010 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823014 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823017 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823020 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823023 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823026 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823029 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823032 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823037 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823040 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823043 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823045 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823048 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823051 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823054 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823057 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823059 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:06.999872 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:06.823062 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.823068 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.823231 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.825323 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.826798 2565 server.go:1019] "Starting client certificate rotation" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.826907 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.826940 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.829913 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.831140 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.842560 2565 log.go:25] "Validated CRI v1 runtime API" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.845839 2565 log.go:25] "Validated CRI v1 image API" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.847614 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 02:45:07.000648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.851860 2565 fs.go:135] Filesystem UUIDs: map[634f1e5a-2be7-4433-9d0b-219f58214420:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7c3335ee-5f42-4ec8-b781-8767b540bd7d:/dev/nvme0n1p3] Jul 20 02:45:07.001196 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.851877 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.857392 2565 manager.go:216] Machine: {Timestamp:2026-07-20 02:45:06.855907084 +0000 UTC m=+1.731708788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103611 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec267817778077286dbdf38ccbee1bbd SystemUUID:ec267817-7780-7728-6dbd-f38ccbee1bbd BootID:ae916269-4bdd-4bd9-8d7a-a0af23d49676 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:48:cd:c0:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:48:cd:c0:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ab:de:82:58:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.857512 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.857638 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.860750 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.860775 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.861407 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.861419 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 02:45:07.001255 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.861427 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.862687 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.864090 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.866018 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868124 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868234 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868265 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868275 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868286 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868295 2565 kubelet.go:398] "Adding apiserver pod source" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868306 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868717 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.868735 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.871849 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.874111 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.874141 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875860 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875877 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875886 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875895 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875905 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875911 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875920 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875925 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875932 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875938 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875952 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.875958 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.876639 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.876652 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.879030 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.879085 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.879093 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.880524 2565 server.go:1268] "Started kubelet" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.880599 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.880626 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.880696 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.883262 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.889208 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 02:45:07.001708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.889372 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-668bp" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.889834 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.889410 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-38.ec2.internal.18c3df6613c9b9fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-38.ec2.internal,UID:ip-10-0-134-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-38.ec2.internal,},FirstTimestamp:2026-07-20 02:45:06.880494075 +0000 UTC m=+1.756295779,LastTimestamp:2026-07-20 02:45:06.880494075 +0000 UTC m=+1.756295779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-38.ec2.internal,}" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894031 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894041 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.894139 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894682 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894698 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894712 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-668bp" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894819 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894860 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894869 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.894999 2565 factory.go:55] Registering systemd factory Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895016 2565 factory.go:223] Registration of the systemd container factory successfully Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895246 2565 factory.go:153] Registering CRI-O factory Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895261 2565 factory.go:223] Registration of the crio container factory successfully Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895360 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895382 2565 factory.go:103] Registering Raw factory Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895397 2565 manager.go:1196] Started watching for new ooms in manager Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.895752 2565 manager.go:349] Starting recovery of all containers Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.906384 2565 manager.go:354] Recovery completed Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.907218 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.910532 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-38.ec2.internal\" not found" node="ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.910952 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.912926 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.912959 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.912977 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.913508 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.913520 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.913520 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.913556 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.914223 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.914248 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:06.914259 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.003310 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:06.994444 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.007151 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.007136 2565 policy_none.go:49] "None policy: Start" Jul 20 02:45:07.007221 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.007157 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 02:45:07.007221 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.007171 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 02:45:07.009389 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.009362 2565 policy_none.go:47] "Start" Jul 20 02:45:07.060998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.060977 2565 manager.go:337] "Starting Device Plugin manager" Jul 20 02:45:07.061103 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.061016 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 02:45:07.061103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061028 2565 server.go:87] "Starting device plugin registration server" Jul 20 02:45:07.061310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061298 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 02:45:07.061361 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061314 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 02:45:07.061457 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061414 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 02:45:07.061537 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061521 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 02:45:07.061537 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.061537 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 02:45:07.062098 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.062079 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 02:45:07.062189 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.062127 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.161990 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.161952 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.164257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.163073 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.164257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.163103 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.164257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.163115 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.164257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.163149 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.171862 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.171846 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.171914 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.171870 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-38.ec2.internal\": node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.183200 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.183169 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 02:45:07.184571 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.184546 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 02:45:07.184671 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.184585 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 02:45:07.184671 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.184608 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 02:45:07.184671 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.184646 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 02:45:07.187109 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.187088 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.189022 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.189002 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.284948 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.284842 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal"] Jul 20 02:45:07.285116 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.284960 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.286072 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.286052 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.286167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.286088 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.286167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.286103 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.288544 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.288530 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.288697 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.288670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.288753 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.288705 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.289090 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.289070 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.289359 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289340 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.289359 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289358 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.289556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289385 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.289556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289408 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.289556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289521 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.289556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.289535 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.291829 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.291815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.291885 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.291844 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:07.292630 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.292609 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:07.292709 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.292640 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:07.292709 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.292656 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:07.297156 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.297134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.297259 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.297166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.297259 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.297193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73ae218cafc3c7d005fc46d807fa2771-config\") pod \"kube-apiserver-proxy-ip-10-0-134-38.ec2.internal\" (UID: \"73ae218cafc3c7d005fc46d807fa2771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.306019 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.305999 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-38.ec2.internal\" not found" node="ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.309569 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.309550 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-38.ec2.internal\" not found" node="ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.389205 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.389175 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.398254 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73ae218cafc3c7d005fc46d807fa2771-config\") pod \"kube-apiserver-proxy-ip-10-0-134-38.ec2.internal\" (UID: \"73ae218cafc3c7d005fc46d807fa2771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.398342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.398342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.398342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73ae218cafc3c7d005fc46d807fa2771-config\") pod \"kube-apiserver-proxy-ip-10-0-134-38.ec2.internal\" (UID: \"73ae218cafc3c7d005fc46d807fa2771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.398504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.398504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.398394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48f06778756ff37380f6a5ec50f29c37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal\" (UID: \"48f06778756ff37380f6a5ec50f29c37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.496099 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.496049 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.601513 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.601436 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-38.ec2.internal\" not found" Jul 20 02:45:07.609167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.609141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.612550 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.612530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.692654 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.692622 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.694777 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.694754 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.706302 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.706270 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 02:45:07.708539 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.708525 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" Jul 20 02:45:07.719143 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.719125 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 02:45:07.827129 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.827097 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 02:45:07.827299 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.827279 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 02:45:07.827379 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.827291 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 02:45:07.827379 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.827311 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 02:45:07.868850 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.868769 2565 apiserver.go:52] "Watching apiserver" Jul 20 02:45:07.884218 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.884182 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 02:45:07.885228 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.885204 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-dv9tw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7","openshift-dns/node-resolver-gb6bp","openshift-multus/multus-mp6b8","openshift-network-diagnostics/network-check-target-v4smd","openshift-ovn-kubernetes/ovnkube-node-lnzx5","kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal","openshift-cluster-node-tuning-operator/tuned-25gwf","openshift-image-registry/node-ca-pl4t4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal","openshift-multus/multus-additional-cni-plugins-lkr6c","openshift-multus/network-metrics-daemon-2vxkh","openshift-network-operator/iptables-alerter-t56mt","kube-system/global-pull-secret-syncer-mm7wt"] Jul 20 02:45:07.887912 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.887894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:07.889817 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.889777 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qvv76\"" Jul 20 02:45:07.889983 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.889970 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 02:45:07.890107 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.890088 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 02:45:07.890276 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.890264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.893252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.893216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:07.893698 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.893677 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 02:45:07.893785 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.893683 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.893846 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.893682 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.893911 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.893887 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ttgmd\"" Jul 20 02:45:07.894473 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.894456 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 02:45:07.895162 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.895138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.895510 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.895482 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.895800 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.895783 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.895943 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.895928 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8hgtf\"" Jul 20 02:45:07.897019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897004 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2bvlw\"" Jul 20 02:45:07.897413 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.897413 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897393 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 02:45:07.897576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897435 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:07.897576 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.897500 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:07.897682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897643 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 02:45:07.897792 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.897775 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.900049 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.900030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:07.900145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.900102 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 02:40:07 +0000 UTC" deadline="2028-01-23 09:27:09.177956977 +0000 UTC" Jul 20 02:45:07.900203 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.900146 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13254h42m1.277814132s" Jul 20 02:45:07.901939 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.901922 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.902551 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902527 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 02:45:07.902641 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 02:45:07.902741 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:07.902784 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902762 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 02:45:07.902823 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 02:45:07.902926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vpwgg\"" Jul 20 02:45:07.903011 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.902943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.904173 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrh6\" (UniqueName: \"kubernetes.io/projected/eaeb2db2-e90d-4c34-884e-41f944356963-kube-api-access-6lrh6\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-registration-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-device-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-sys-fs\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904362 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904362 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52576f0b-79a1-48bc-8438-ba81338d0429-konnectivity-ca\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:07.904362 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-socket-dir-parent\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904405 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-bin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-kubelet\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8psr\" (UniqueName: \"kubernetes.io/projected/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-kube-api-access-n8psr\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-multus-certs\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-tmp-dir\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-cni-binary-copy\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-hostroot\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5pt\" (UniqueName: \"kubernetes.io/projected/5d2ff516-5993-40f4-a372-e115df40a5b1-kube-api-access-ql5pt\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-k8s-cni-cncf-io\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52576f0b-79a1-48bc-8438-ba81338d0429-agent-certs\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-netns\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-cnibin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-socket-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-os-release\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.904752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-multus\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-multus-daemon-config\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-hosts-file\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904848 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-system-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-conf-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904916 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-etc-kubernetes\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.904975 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.905032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99tmw\"" Jul 20 02:45:07.905213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.905176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:07.907354 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.907337 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.907429 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.907363 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 02:45:07.907429 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.907391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.907600 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.907585 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 02:45:07.907963 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.907945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:07.908389 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.908370 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzrcz\"" Jul 20 02:45:07.909957 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.909937 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzlwp\"" Jul 20 02:45:07.910059 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.909985 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 02:45:07.910059 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.909985 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 02:45:07.910386 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.910371 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:07.910466 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.910442 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:07.912845 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.912829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:07.915131 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.915111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.915208 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.915117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.915208 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.915189 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 02:45:07.915404 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.915391 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zb46q\"" Jul 20 02:45:07.915447 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.915401 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:07.915481 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:07.915449 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:07.930827 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.930804 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8tq98" Jul 20 02:45:07.940251 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.940230 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8tq98" Jul 20 02:45:07.996143 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:07.996122 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 02:45:08.005259 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-cni-binary-copy\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-hostroot\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5pt\" (UniqueName: \"kubernetes.io/projected/5d2ff516-5993-40f4-a372-e115df40a5b1-kube-api-access-ql5pt\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-k8s-cni-cncf-io\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52576f0b-79a1-48bc-8438-ba81338d0429-agent-certs\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-slash\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.005381 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-kubernetes\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-host\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-hostroot\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea855527-0972-4f91-8790-fb5b6217bd38-host\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-k8s-cni-cncf-io\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-cnibin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-socket-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-kubelet\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.005631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-cnibin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-etc-kubernetes\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-registration-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-socket-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-sys-fs\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-bin\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-script-lib\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/4fcacf48-1d67-4537-ad9e-64714f2856b1-kube-api-access-vrkrr\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005810 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-registration-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af453b92-ade4-44dd-aa09-a10f221a5732-host-slash\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-sys-fs\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006191 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-etc-kubernetes\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-systemd-units\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-systemd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005966 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.005990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-ovn\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysconfig\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-bin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006114 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-node-log\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-sys\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.006984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-bin\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-var-lib-kubelet\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qpm\" (UniqueName: \"kubernetes.io/projected/0cbf1d55-411c-48ce-b16f-33b6adf859e7-kube-api-access-h2qpm\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcpf\" (UniqueName: \"kubernetes.io/projected/af453b92-ade4-44dd-aa09-a10f221a5732-kube-api-access-whcpf\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8psr\" (UniqueName: \"kubernetes.io/projected/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-kube-api-access-n8psr\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-netd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-conf\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-run\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-tmp-dir\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6m6\" (UniqueName: \"kubernetes.io/projected/ea855527-0972-4f91-8790-fb5b6217bd38-kube-api-access-jq6m6\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trr9z\" (UniqueName: \"kubernetes.io/projected/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-kube-api-access-trr9z\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-kubelet-config\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.007773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006814 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-cni-binary-copy\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-log-socket\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-lib-modules\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkccn\" (UniqueName: \"kubernetes.io/projected/913676fd-3375-44c5-ba7f-b6f329121885-kube-api-access-nkccn\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea855527-0972-4f91-8790-fb5b6217bd38-serviceca\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-netns\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-etc-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-tmp\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-tmp-dir\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-dbus\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.006982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-netns\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-os-release\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-multus\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-multus-daemon-config\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.008458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-os-release\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007091 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-cni-multus\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-systemd\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af453b92-ade4-44dd-aa09-a10f221a5732-iptables-alerter-script\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cnibin\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-hosts-file\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-system-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-conf-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrh6\" (UniqueName: \"kubernetes.io/projected/eaeb2db2-e90d-4c34-884e-41f944356963-kube-api-access-6lrh6\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-device-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-netns\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-conf-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.008903 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-hosts-file\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-system-cni-dir\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d2ff516-5993-40f4-a372-e115df40a5b1-device-dir\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-config\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eaeb2db2-e90d-4c34-884e-41f944356963-multus-daemon-config\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52576f0b-79a1-48bc-8438-ba81338d0429-konnectivity-ca\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-var-lib-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-modprobe-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-env-overrides\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-etc-tuned\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-socket-dir-parent\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-kubelet\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-multus-socket-dir-parent\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-multus-certs\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-run-multus-certs\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaeb2db2-e90d-4c34-884e-41f944356963-host-var-lib-kubelet\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.007903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-os-release\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.008053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52576f0b-79a1-48bc-8438-ba81338d0429-konnectivity-ca\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:08.009998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.009811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52576f0b-79a1-48bc-8438-ba81338d0429-agent-certs\") pod \"konnectivity-agent-dv9tw\" (UID: \"52576f0b-79a1-48bc-8438-ba81338d0429\") " pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:08.012944 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.012922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5pt\" (UniqueName: \"kubernetes.io/projected/5d2ff516-5993-40f4-a372-e115df40a5b1-kube-api-access-ql5pt\") pod \"aws-ebs-csi-driver-node-66qj7\" (UID: \"5d2ff516-5993-40f4-a372-e115df40a5b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.016815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.016781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8psr\" (UniqueName: \"kubernetes.io/projected/b41a9a75-9ff7-4235-a372-66fc6b1dc25e-kube-api-access-n8psr\") pod \"node-resolver-gb6bp\" (UID: \"b41a9a75-9ff7-4235-a372-66fc6b1dc25e\") " pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.017749 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.017732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrh6\" (UniqueName: \"kubernetes.io/projected/eaeb2db2-e90d-4c34-884e-41f944356963-kube-api-access-6lrh6\") pod \"multus-mp6b8\" (UID: \"eaeb2db2-e90d-4c34-884e-41f944356963\") " pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-bin\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-script-lib\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/4fcacf48-1d67-4537-ad9e-64714f2856b1-kube-api-access-vrkrr\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-bin\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af453b92-ade4-44dd-aa09-a10f221a5732-host-slash\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-systemd-units\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-systemd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.109983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-ovn\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysconfig\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-node-log\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-systemd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-sys\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.111674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af453b92-ade4-44dd-aa09-a10f221a5732-host-slash\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-var-lib-kubelet\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-sys\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qpm\" (UniqueName: \"kubernetes.io/projected/0cbf1d55-411c-48ce-b16f-33b6adf859e7-kube-api-access-h2qpm\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcpf\" (UniqueName: \"kubernetes.io/projected/af453b92-ade4-44dd-aa09-a10f221a5732-kube-api-access-whcpf\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-netd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-script-lib\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-conf\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-run\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysconfig\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110763 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-run-ovn\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.112703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6m6\" (UniqueName: \"kubernetes.io/projected/ea855527-0972-4f91-8790-fb5b6217bd38-kube-api-access-jq6m6\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trr9z\" (UniqueName: \"kubernetes.io/projected/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-kube-api-access-trr9z\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-kubelet-config\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-conf\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-log-socket\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.110992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-lib-modules\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkccn\" (UniqueName: \"kubernetes.io/projected/913676fd-3375-44c5-ba7f-b6f329121885-kube-api-access-nkccn\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea855527-0972-4f91-8790-fb5b6217bd38-serviceca\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-etc-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-cni-netd\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-kubelet-config\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-var-lib-kubelet\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-log-socket\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-run\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.113511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-etc-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-node-log\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-tmp\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-systemd-units\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-lib-modules\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112051 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-dbus\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.111860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-dbus\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea855527-0972-4f91-8790-fb5b6217bd38-serviceca\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-systemd\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112230 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-systemd\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af453b92-ade4-44dd-aa09-a10f221a5732-iptables-alerter-script\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cnibin\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.114145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-netns\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112364 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-cnibin\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-config\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.112522 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-var-lib-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.112720 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.612697069 +0000 UTC m=+3.488498760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-netns\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-modprobe-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-var-lib-openvswitch\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-env-overrides\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-etc-tuned\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.112980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af453b92-ade4-44dd-aa09-a10f221a5732-iptables-alerter-script\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.114818 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113089 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-modprobe-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-os-release\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-slash\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-kubernetes\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-host\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-env-overrides\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-slash\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-os-release\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea855527-0972-4f91-8790-fb5b6217bd38-host\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-host\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovnkube-config\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-system-cni-dir\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.113983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea855527-0972-4f91-8790-fb5b6217bd38-host\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-kubernetes\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-kubelet\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.114206 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fcacf48-1d67-4537-ad9e-64714f2856b1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.114277 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.614259917 +0000 UTC m=+3.490061622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcacf48-1d67-4537-ad9e-64714f2856b1-host-kubelet\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.115905 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/913676fd-3375-44c5-ba7f-b6f329121885-etc-sysctl-d\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.116341 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-tmp\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.116341 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.114766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.116341 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.115832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/913676fd-3375-44c5-ba7f-b6f329121885-etc-tuned\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.118953 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.118893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trr9z\" (UniqueName: \"kubernetes.io/projected/dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d-kube-api-access-trr9z\") pod \"multus-additional-cni-plugins-lkr6c\" (UID: \"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d\") " pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.119493 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.119470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkrr\" (UniqueName: \"kubernetes.io/projected/4fcacf48-1d67-4537-ad9e-64714f2856b1-kube-api-access-vrkrr\") pod \"ovnkube-node-lnzx5\" (UID: \"4fcacf48-1d67-4537-ad9e-64714f2856b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.119800 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.119784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qpm\" (UniqueName: \"kubernetes.io/projected/0cbf1d55-411c-48ce-b16f-33b6adf859e7-kube-api-access-h2qpm\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.121607 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.121589 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:08.121607 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.121608 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:08.121804 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.121618 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.121804 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.121670 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.621657417 +0000 UTC m=+3.497459112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.122870 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.122846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6m6\" (UniqueName: \"kubernetes.io/projected/ea855527-0972-4f91-8790-fb5b6217bd38-kube-api-access-jq6m6\") pod \"node-ca-pl4t4\" (UID: \"ea855527-0972-4f91-8790-fb5b6217bd38\") " pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.123672 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.123653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkccn\" (UniqueName: \"kubernetes.io/projected/913676fd-3375-44c5-ba7f-b6f329121885-kube-api-access-nkccn\") pod \"tuned-25gwf\" (UID: \"913676fd-3375-44c5-ba7f-b6f329121885\") " pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.123923 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.123903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcpf\" (UniqueName: \"kubernetes.io/projected/af453b92-ade4-44dd-aa09-a10f221a5732-kube-api-access-whcpf\") pod \"iptables-alerter-t56mt\" (UID: \"af453b92-ade4-44dd-aa09-a10f221a5732\") " pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.213685 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.213642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:08.230306 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.230260 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:08.231979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.231952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" Jul 20 02:45:08.250768 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.250684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gb6bp" Jul 20 02:45:08.250768 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.250706 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f06778756ff37380f6a5ec50f29c37.slice/crio-fc2679cefd8fd4f15db55a2fa50d560322f7c5176365a0c4c4ef459aa7310a79 WatchSource:0}: Error finding container fc2679cefd8fd4f15db55a2fa50d560322f7c5176365a0c4c4ef459aa7310a79: Status 404 returned error can't find the container with id fc2679cefd8fd4f15db55a2fa50d560322f7c5176365a0c4c4ef459aa7310a79 Jul 20 02:45:08.251200 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.251165 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52576f0b_79a1_48bc_8438_ba81338d0429.slice/crio-fb3cb92fa9128528f0fc0c8ecb5d253246e6e6296ad8feaef9717cb09228c98d WatchSource:0}: Error finding container fb3cb92fa9128528f0fc0c8ecb5d253246e6e6296ad8feaef9717cb09228c98d: Status 404 returned error can't find the container with id fb3cb92fa9128528f0fc0c8ecb5d253246e6e6296ad8feaef9717cb09228c98d Jul 20 02:45:08.254799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.254645 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mp6b8" Jul 20 02:45:08.256918 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.256896 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:45:08.258077 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.258051 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ae218cafc3c7d005fc46d807fa2771.slice/crio-cd600b14db7d546d712bf4f28cbb076a22e6c1d6f55285c391addad4dec478dc WatchSource:0}: Error finding container cd600b14db7d546d712bf4f28cbb076a22e6c1d6f55285c391addad4dec478dc: Status 404 returned error can't find the container with id cd600b14db7d546d712bf4f28cbb076a22e6c1d6f55285c391addad4dec478dc Jul 20 02:45:08.260371 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.260345 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2ff516_5993_40f4_a372_e115df40a5b1.slice/crio-bae99238395628a343cde17f65e3dd1dcd2de8352cc2ddc0bb436b673b3bfa70 WatchSource:0}: Error finding container bae99238395628a343cde17f65e3dd1dcd2de8352cc2ddc0bb436b673b3bfa70: Status 404 returned error can't find the container with id bae99238395628a343cde17f65e3dd1dcd2de8352cc2ddc0bb436b673b3bfa70 Jul 20 02:45:08.264950 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.264923 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41a9a75_9ff7_4235_a372_66fc6b1dc25e.slice/crio-c599c6857a10cac35e349d3c36cb9cde5a6fa42417425ad6a30ac350815494e4 WatchSource:0}: Error finding container c599c6857a10cac35e349d3c36cb9cde5a6fa42417425ad6a30ac350815494e4: Status 404 returned error can't find the container with id c599c6857a10cac35e349d3c36cb9cde5a6fa42417425ad6a30ac350815494e4 Jul 20 02:45:08.266825 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.266792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:08.266983 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.266950 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaeb2db2_e90d_4c34_884e_41f944356963.slice/crio-1b841dfcc96f9392e099ba1fd183fc145ac5c9c061148b83af726f12042b558a WatchSource:0}: Error finding container 1b841dfcc96f9392e099ba1fd183fc145ac5c9c061148b83af726f12042b558a: Status 404 returned error can't find the container with id 1b841dfcc96f9392e099ba1fd183fc145ac5c9c061148b83af726f12042b558a Jul 20 02:45:08.276675 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.276656 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcacf48_1d67_4537_ad9e_64714f2856b1.slice/crio-5423fc5d6f856537778da51584aeea49b3a47f5969ddc66f5d67e4f38b258a18 WatchSource:0}: Error finding container 5423fc5d6f856537778da51584aeea49b3a47f5969ddc66f5d67e4f38b258a18: Status 404 returned error can't find the container with id 5423fc5d6f856537778da51584aeea49b3a47f5969ddc66f5d67e4f38b258a18 Jul 20 02:45:08.281975 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.281955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25gwf" Jul 20 02:45:08.286341 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.286323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pl4t4" Jul 20 02:45:08.291898 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.291085 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913676fd_3375_44c5_ba7f_b6f329121885.slice/crio-7d8dac9c99fd888660d6d544548f6c6ca7dfd564c98305ab3c0ad93beb542224 WatchSource:0}: Error finding container 7d8dac9c99fd888660d6d544548f6c6ca7dfd564c98305ab3c0ad93beb542224: Status 404 returned error can't find the container with id 7d8dac9c99fd888660d6d544548f6c6ca7dfd564c98305ab3c0ad93beb542224 Jul 20 02:45:08.297638 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.297612 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea855527_0972_4f91_8790_fb5b6217bd38.slice/crio-4c5b8c5a2dd56b9024ca1909098bf8080a2689dfe83a9d763787b5223d65898c WatchSource:0}: Error finding container 4c5b8c5a2dd56b9024ca1909098bf8080a2689dfe83a9d763787b5223d65898c: Status 404 returned error can't find the container with id 4c5b8c5a2dd56b9024ca1909098bf8080a2689dfe83a9d763787b5223d65898c Jul 20 02:45:08.302869 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.302852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" Jul 20 02:45:08.309135 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.309117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t56mt" Jul 20 02:45:08.312416 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.312394 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc4af98_d2b1_4db8_9acd_17e9a8c3b99d.slice/crio-3eb73a93ffb3a5f979fbf61c9634a09b52372f5fa202e09dad11b94a1e96798a WatchSource:0}: Error finding container 3eb73a93ffb3a5f979fbf61c9634a09b52372f5fa202e09dad11b94a1e96798a: Status 404 returned error can't find the container with id 3eb73a93ffb3a5f979fbf61c9634a09b52372f5fa202e09dad11b94a1e96798a Jul 20 02:45:08.317557 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:08.317539 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf453b92_ade4_44dd_aa09_a10f221a5732.slice/crio-1a2d068857eeb0f8ef29e3d1acb98b1d8f2fcb8f849b66eb3930368b5afe95b4 WatchSource:0}: Error finding container 1a2d068857eeb0f8ef29e3d1acb98b1d8f2fcb8f849b66eb3930368b5afe95b4: Status 404 returned error can't find the container with id 1a2d068857eeb0f8ef29e3d1acb98b1d8f2fcb8f849b66eb3930368b5afe95b4 Jul 20 02:45:08.319425 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.319405 2565 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-t56mt_openshift-network-operator(af453b92-ade4-44dd-aa09-a10f221a5732): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 20 02:45:08.319493 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.319437 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-t56mt" podUID="af453b92-ade4-44dd-aa09-a10f221a5732" Jul 20 02:45:08.619176 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.619152 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:08.619344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.619202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:08.619344 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.619300 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.619344 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.619315 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.619464 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.619377 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:09.619355466 +0000 UTC m=+4.495157156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.619464 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.619396 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:09.619386825 +0000 UTC m=+4.495188515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.720664 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.720628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:08.720846 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.720817 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:08.720846 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.720839 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:08.720961 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.720855 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.720961 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:08.720920 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:09.720898487 +0000 UTC m=+4.596700180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.944247 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.944127 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 02:40:08 +0000 UTC" deadline="2028-02-25 10:26:40.145999059 +0000 UTC" Jul 20 02:45:08.944247 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:08.944195 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14047h41m31.201808925s" Jul 20 02:45:09.038165 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.038132 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:09.188508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.188437 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:09.188738 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.188578 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:09.213207 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.213098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mp6b8" event={"ID":"eaeb2db2-e90d-4c34-884e-41f944356963","Type":"ContainerStarted","Data":"1b841dfcc96f9392e099ba1fd183fc145ac5c9c061148b83af726f12042b558a"} Jul 20 02:45:09.223332 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.223236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv9tw" event={"ID":"52576f0b-79a1-48bc-8438-ba81338d0429","Type":"ContainerStarted","Data":"fb3cb92fa9128528f0fc0c8ecb5d253246e6e6296ad8feaef9717cb09228c98d"} Jul 20 02:45:09.231204 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.231169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" event={"ID":"48f06778756ff37380f6a5ec50f29c37","Type":"ContainerStarted","Data":"fc2679cefd8fd4f15db55a2fa50d560322f7c5176365a0c4c4ef459aa7310a79"} Jul 20 02:45:09.253431 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.253378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pl4t4" event={"ID":"ea855527-0972-4f91-8790-fb5b6217bd38","Type":"ContainerStarted","Data":"4c5b8c5a2dd56b9024ca1909098bf8080a2689dfe83a9d763787b5223d65898c"} Jul 20 02:45:09.274547 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.274510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25gwf" event={"ID":"913676fd-3375-44c5-ba7f-b6f329121885","Type":"ContainerStarted","Data":"7d8dac9c99fd888660d6d544548f6c6ca7dfd564c98305ab3c0ad93beb542224"} Jul 20 02:45:09.280708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.280673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gb6bp" event={"ID":"b41a9a75-9ff7-4235-a372-66fc6b1dc25e","Type":"ContainerStarted","Data":"c599c6857a10cac35e349d3c36cb9cde5a6fa42417425ad6a30ac350815494e4"} Jul 20 02:45:09.292648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.292577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" event={"ID":"5d2ff516-5993-40f4-a372-e115df40a5b1","Type":"ContainerStarted","Data":"bae99238395628a343cde17f65e3dd1dcd2de8352cc2ddc0bb436b673b3bfa70"} Jul 20 02:45:09.308095 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.308037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" event={"ID":"73ae218cafc3c7d005fc46d807fa2771","Type":"ContainerStarted","Data":"cd600b14db7d546d712bf4f28cbb076a22e6c1d6f55285c391addad4dec478dc"} Jul 20 02:45:09.332166 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.332129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t56mt" event={"ID":"af453b92-ade4-44dd-aa09-a10f221a5732","Type":"ContainerStarted","Data":"1a2d068857eeb0f8ef29e3d1acb98b1d8f2fcb8f849b66eb3930368b5afe95b4"} Jul 20 02:45:09.333906 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.333804 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:09.339211 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.339184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerStarted","Data":"3eb73a93ffb3a5f979fbf61c9634a09b52372f5fa202e09dad11b94a1e96798a"} Jul 20 02:45:09.342037 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.342006 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e83687a3e51e78c37199d4d4504faf8bfeec5d995d123db950ef57fe67957539\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-t56mt" podUID="af453b92-ade4-44dd-aa09-a10f221a5732" Jul 20 02:45:09.347478 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.347450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"5423fc5d6f856537778da51584aeea49b3a47f5969ddc66f5d67e4f38b258a18"} Jul 20 02:45:09.627821 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.627755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:09.628017 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.627833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:09.628017 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.627968 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:09.628136 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.628034 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:11.628015684 +0000 UTC m=+6.503817377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:09.628584 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.628513 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:09.628584 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.628583 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:11.62856424 +0000 UTC m=+6.504365945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:09.729180 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.729137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:09.729407 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.729366 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:09.729479 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.729439 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:09.729479 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.729459 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:09.729599 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:09.729539 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:11.729519231 +0000 UTC m=+6.605320920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:09.945168 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.945098 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 02:40:08 +0000 UTC" deadline="2028-01-31 14:33:03.429608212 +0000 UTC" Jul 20 02:45:09.945168 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:09.945165 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13451h47m53.484448142s" Jul 20 02:45:10.170829 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:10.170750 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:10.185642 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:10.185592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:10.185803 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:10.185711 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:10.186139 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:10.186121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:10.186279 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:10.186230 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:11.186536 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:11.186502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:11.187124 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.186642 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:11.646021 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:11.645938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:11.646021 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:11.646009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:11.646246 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.646140 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:11.646246 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.646223 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:15.646180996 +0000 UTC m=+10.521982691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:11.646611 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.646527 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:11.646611 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.646590 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:15.64657243 +0000 UTC m=+10.522374123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:11.746772 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:11.746729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:11.746954 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.746945 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:11.747015 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.746965 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:11.747015 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.746978 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:11.747178 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:11.747033 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:15.74701503 +0000 UTC m=+10.622816733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:12.185699 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:12.185669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:12.185996 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:12.185820 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:12.186278 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:12.185629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:12.186278 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:12.186237 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:13.186219 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:13.185770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:13.186219 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:13.185916 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:14.185076 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:14.184824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:14.185076 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:14.184941 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:14.185076 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:14.184961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:14.185076 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:14.185040 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:15.185713 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:15.185674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:15.186204 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.185820 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:15.679408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:15.679314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:15.679408 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.679401 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:15.679622 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.679471 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:23.679450133 +0000 UTC m=+18.555251821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:15.679622 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.679491 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:15.679622 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.679547 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:23.679529435 +0000 UTC m=+18.555331128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:15.679622 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:15.679406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:15.780466 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:15.780381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:15.780654 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.780572 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:15.780654 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.780595 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:15.780654 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.780607 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:15.780826 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:15.780662 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:23.780644018 +0000 UTC m=+18.656445721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:16.185832 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:16.185637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:16.185832 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:16.185762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:16.186313 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:16.185834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:16.186313 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:16.185961 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:17.186835 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:17.186322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:17.186835 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:17.186443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:18.185709 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:18.185671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:18.185891 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:18.185671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:18.185891 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:18.185812 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:18.186009 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:18.185900 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:19.185234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:19.185195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:19.185723 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:19.185349 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:20.185861 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:20.185819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:20.185861 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:20.185842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:20.186407 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:20.186044 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:20.186407 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:20.186172 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:21.185088 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:21.185052 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:21.185269 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:21.185174 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:22.185139 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:22.185105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:22.185660 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:22.185104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:22.185660 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:22.185254 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:22.185660 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:22.185360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:23.185458 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:23.185421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:23.185888 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.185570 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:23.747866 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:23.747782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:23.747866 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:23.747840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:23.748071 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.747932 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:23.748071 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.747942 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:23.748071 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.747986 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.747968378 +0000 UTC m=+34.623770069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:23.748071 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.748006 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.747995681 +0000 UTC m=+34.623797371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:23.848149 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:23.848114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:23.848353 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.848240 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:23.848353 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.848265 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:23.848353 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.848281 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:23.848353 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:23.848354 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.848338179 +0000 UTC m=+34.724139867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:24.185185 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:24.185110 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:24.185185 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:24.185127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:24.185405 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:24.185253 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:24.185405 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:24.185379 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:25.185727 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:25.185694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:25.186177 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:25.185804 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:26.185231 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:26.185199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:26.185401 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:26.185331 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:26.185401 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:26.185363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:26.185501 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:26.185422 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:27.186525 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:27.186484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:27.186957 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:27.186590 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:28.185228 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.185206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:28.185361 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.185240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:28.185361 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:28.185327 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:28.185474 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:28.185451 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:28.389227 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.389200 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:45:28.389716 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.389689 2565 generic.go:356] "Generic (PLEG): container finished" podID="4fcacf48-1d67-4537-ad9e-64714f2856b1" containerID="a36f6fce65cce6c73ba4655d486f55ad3f02e948a43d68cb72753e0b3a68a0b5" exitCode=1 Jul 20 02:45:28.389779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.389749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"6aad65699331bd2ff7157b4691165fc43788e1d413456c3f520f131197eeea65"} Jul 20 02:45:28.389878 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.389863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerDied","Data":"a36f6fce65cce6c73ba4655d486f55ad3f02e948a43d68cb72753e0b3a68a0b5"} Jul 20 02:45:28.389935 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.389917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"0d9c320785c21241dab926bb8b4288281ad00a0a2b8539440137d3a43d3c06ff"} Jul 20 02:45:28.391409 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.391387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mp6b8" event={"ID":"eaeb2db2-e90d-4c34-884e-41f944356963","Type":"ContainerStarted","Data":"df4ce284b2452f3eb708fee7065700f7cd7e68084cf94cf2bc54c34ddad6cde6"} Jul 20 02:45:28.393225 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.393202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25gwf" event={"ID":"913676fd-3375-44c5-ba7f-b6f329121885","Type":"ContainerStarted","Data":"64837f59ae38be9b8566d256433fc7958bfc3b84b432bef975a713598052c490"} Jul 20 02:45:28.394676 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.394652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" event={"ID":"73ae218cafc3c7d005fc46d807fa2771","Type":"ContainerStarted","Data":"81d1d2d0f2af44e7e672cca17f11e264eeda5f53f2809a112f7ce0dde066494a"} Jul 20 02:45:28.406716 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.406678 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mp6b8" podStartSLOduration=1.9508192370000002 podStartE2EDuration="21.406666366s" totalImagesPullingTime="19.455847129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.40620565 +0000 UTC m=+23.282007360" watchObservedRunningTime="2026-07-20 02:45:28.406666366 +0000 UTC m=+23.282468076" Jul 20 02:45:28.431562 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.431521 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-38.ec2.internal" podStartSLOduration=21.431507792 podStartE2EDuration="21.431507792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.417422456 +0000 UTC m=+23.293224167" watchObservedRunningTime="2026-07-20 02:45:28.431507792 +0000 UTC m=+23.307309523" Jul 20 02:45:28.431739 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:28.431581 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-25gwf" podStartSLOduration=2.03574725 podStartE2EDuration="21.431577271s" totalImagesPullingTime="19.395830021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.431218186 +0000 UTC m=+23.307019896" watchObservedRunningTime="2026-07-20 02:45:28.431577271 +0000 UTC m=+23.307378975" Jul 20 02:45:29.185357 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.185326 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:29.185513 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:29.185434 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:29.398089 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.398051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gb6bp" event={"ID":"b41a9a75-9ff7-4235-a372-66fc6b1dc25e","Type":"ContainerStarted","Data":"32dd5c76dfb196e09ccda8d490749d6f7562ebbf84df09842812294e699e0f09"} Jul 20 02:45:29.399431 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.399408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" event={"ID":"5d2ff516-5993-40f4-a372-e115df40a5b1","Type":"ContainerStarted","Data":"6c8012cd9a3d045712077497b18afa6fb1de1166e51a1f0a8a1548158bd94d1c"} Jul 20 02:45:29.400678 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.400654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t56mt" event={"ID":"af453b92-ade4-44dd-aa09-a10f221a5732","Type":"ContainerStarted","Data":"37fc789f8c80513c3b5aa492ead4752f7a261ef6ce3d466333536d391d00a023"} Jul 20 02:45:29.402112 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.402091 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="4a5bb686ff94b1095583220ae57903a12f0fd376eb5986557209ec99d5048b09" exitCode=0 Jul 20 02:45:29.402210 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.402160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"4a5bb686ff94b1095583220ae57903a12f0fd376eb5986557209ec99d5048b09"} Jul 20 02:45:29.404920 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.404903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:45:29.405342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.405320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"c6602104db42e1e63fc2941e59416e02765716f343da6fea90912dfabb2c37dd"} Jul 20 02:45:29.405451 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.405393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"49787bfb5683c6741312efcb5432e4bf818f241c52a3c66109584bbbdebdb179"} Jul 20 02:45:29.405451 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.405445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"6e0c870eec16f0f51ce110ffe6d39ffcf7cef2d86a279b5b9888c2f1816fbcee"} Jul 20 02:45:29.406678 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.406647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv9tw" event={"ID":"52576f0b-79a1-48bc-8438-ba81338d0429","Type":"ContainerStarted","Data":"baef72d9f04274b64efc6fa79f4f6b1def0b8390a4f7fce98fd3b67c3536f796"} Jul 20 02:45:29.407935 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.407913 2565 generic.go:356] "Generic (PLEG): container finished" podID="48f06778756ff37380f6a5ec50f29c37" containerID="0b0367de8cd445cde49af4b597caaef553bb611ca291130f0d73e9e1331e331a" exitCode=0 Jul 20 02:45:29.408016 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.407982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" event={"ID":"48f06778756ff37380f6a5ec50f29c37","Type":"ContainerDied","Data":"0b0367de8cd445cde49af4b597caaef553bb611ca291130f0d73e9e1331e331a"} Jul 20 02:45:29.409403 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.409380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pl4t4" event={"ID":"ea855527-0972-4f91-8790-fb5b6217bd38","Type":"ContainerStarted","Data":"2ca739ba5975799f8b7348af606f24212bbd5c954aa4b3c7e23c9e5ad885d589"} Jul 20 02:45:29.413102 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.413068 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gb6bp" podStartSLOduration=2.991379699 podStartE2EDuration="22.413057692s" totalImagesPullingTime="19.421677993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:29.412930679 +0000 UTC m=+24.288732389" watchObservedRunningTime="2026-07-20 02:45:29.413057692 +0000 UTC m=+24.288859401" Jul 20 02:45:29.426613 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.426569 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t56mt" podStartSLOduration=2.730683041 podStartE2EDuration="22.426555728s" totalImagesPullingTime="19.695872687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-20 02:45:29.425950905 +0000 UTC m=+24.301752617" watchObservedRunningTime="2026-07-20 02:45:29.426555728 +0000 UTC m=+24.302357438" Jul 20 02:45:29.450687 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.450597 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dv9tw" podStartSLOduration=3.018808232 podStartE2EDuration="22.450577769s" totalImagesPullingTime="19.431769537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:29.450514984 +0000 UTC m=+24.326316694" watchObservedRunningTime="2026-07-20 02:45:29.450577769 +0000 UTC m=+24.326379480" Jul 20 02:45:29.464781 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.464723 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pl4t4" podStartSLOduration=3.075515935 podStartE2EDuration="22.464707457s" totalImagesPullingTime="19.389191522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:29.464566152 +0000 UTC m=+24.340367875" watchObservedRunningTime="2026-07-20 02:45:29.464707457 +0000 UTC m=+24.340509167" Jul 20 02:45:29.799404 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.799205 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:29.800002 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:29.799984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:30.085339 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.085306 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 02:45:30.185035 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.185010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:30.185256 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:30.185120 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:30.185256 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.185176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:30.185390 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:30.185278 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:30.414119 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.414041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" event={"ID":"5d2ff516-5993-40f4-a372-e115df40a5b1","Type":"ContainerStarted","Data":"495f6f79bcf50d6fcfea1521eb8bb52b1c4179225e3811a0f0a7d6cd2e5ed4a0"} Jul 20 02:45:30.415986 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.415955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" event={"ID":"48f06778756ff37380f6a5ec50f29c37","Type":"ContainerStarted","Data":"4574de9168d9b3402b2f5878d031656e0b2390390614d12736ae24c83255b5b5"} Jul 20 02:45:30.416640 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.416608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:30.417166 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.417139 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dv9tw" Jul 20 02:45:30.430676 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:30.430623 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-38.ec2.internal" podStartSLOduration=23.430605626 podStartE2EDuration="23.430605626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:30.430134935 +0000 UTC m=+25.305936648" watchObservedRunningTime="2026-07-20 02:45:30.430605626 +0000 UTC m=+25.306407338" Jul 20 02:45:31.072091 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.071983 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T02:45:30.085334378Z","UUID":"a6009afd-2d9a-48b0-8f53-0e9e0c6876c4","Handler":null,"Name":"","Endpoint":""} Jul 20 02:45:31.073764 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.073743 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 02:45:31.073910 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.073785 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 02:45:31.185479 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.185444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:31.185625 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:31.185553 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:31.423036 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.422947 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:45:31.423732 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:31.423701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"81b454ecd7f940f5fd5114d55131d57873ea4275d4e593aae63e60d2bb75e5fa"} Jul 20 02:45:32.185646 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:32.185618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:32.185819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:32.185650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:32.185819 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:32.185720 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:32.185819 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:32.185795 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:32.428234 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:32.428192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" event={"ID":"5d2ff516-5993-40f4-a372-e115df40a5b1","Type":"ContainerStarted","Data":"b7c4c8dbf949f46ecdc06f4cc61882ccf01764bec6c6a0ca143f80b4ddeb9ad6"} Jul 20 02:45:32.443254 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:32.443156 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66qj7" podStartSLOduration=3.546484279 podStartE2EDuration="25.443137099s" totalImagesPullingTime="21.89665282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:32.44289479 +0000 UTC m=+27.318696500" watchObservedRunningTime="2026-07-20 02:45:32.443137099 +0000 UTC m=+27.318938811" Jul 20 02:45:33.185206 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:33.185168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:33.185398 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:33.185323 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:34.185782 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.185538 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:34.186427 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.185574 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:34.186427 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:34.185844 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:34.186427 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:34.185904 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:34.434663 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.434598 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="f49e668d1de6d10817b2d732d62a6d58ea168b3340492c54ca97e6c4fedde7b9" exitCode=0 Jul 20 02:45:34.434847 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.434688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"f49e668d1de6d10817b2d732d62a6d58ea168b3340492c54ca97e6c4fedde7b9"} Jul 20 02:45:34.438344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.438325 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:45:34.438689 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.438670 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"dd5e53a2c98856664921f9c94f4126b19205c425d86a84237fef717fa6f4eb43"} Jul 20 02:45:34.438937 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.438915 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:34.439090 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.439077 2565 scope.go:117] "RemoveContainer" containerID="a36f6fce65cce6c73ba4655d486f55ad3f02e948a43d68cb72753e0b3a68a0b5" Jul 20 02:45:34.454888 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:34.454869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:35.184892 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.184864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:35.185137 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:35.184997 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:35.442682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.442651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerStarted","Data":"00f63fe430069eb9a9e291c460f0f470192b7c09ddbfea545c011a34c13aafda"} Jul 20 02:45:35.446482 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.446465 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:45:35.448511 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.447159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" event={"ID":"4fcacf48-1d67-4537-ad9e-64714f2856b1","Type":"ContainerStarted","Data":"1fcd6ef9ccaaef76e677c73348eadbe5edc9eb421bd6d252bbf593f282ac25c9"} Jul 20 02:45:35.448643 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.448627 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:35.449360 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.448752 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:35.468568 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.468542 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:45:35.484167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.484125 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" podStartSLOduration=8.679075601 podStartE2EDuration="28.484109724s" totalImagesPullingTime="19.805034123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:35.483803842 +0000 UTC m=+30.359605590" watchObservedRunningTime="2026-07-20 02:45:35.484109724 +0000 UTC m=+30.359911434" Jul 20 02:45:35.796118 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.795884 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mm7wt"] Jul 20 02:45:35.796253 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.796135 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:35.796253 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:35.796214 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:35.798821 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.798795 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v4smd"] Jul 20 02:45:35.798962 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.798897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:35.799026 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:35.798990 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:35.799422 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.799401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vxkh"] Jul 20 02:45:35.799547 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:35.799494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:35.799593 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:35.799575 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:36.451076 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:36.451049 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="00f63fe430069eb9a9e291c460f0f470192b7c09ddbfea545c011a34c13aafda" exitCode=0 Jul 20 02:45:36.451523 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:36.451144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"00f63fe430069eb9a9e291c460f0f470192b7c09ddbfea545c011a34c13aafda"} Jul 20 02:45:37.187113 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:37.187021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:37.187360 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:37.187147 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:37.187360 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:37.187201 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:37.187512 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:37.187350 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:38.185612 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:38.185578 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:38.186097 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:38.185710 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:38.457774 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:38.457687 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="c2195db5ab7efb3ed1c0de45d6fdfe4fc4d26b75f2734953aabab9fdff83ce96" exitCode=0 Jul 20 02:45:38.457774 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:38.457746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"c2195db5ab7efb3ed1c0de45d6fdfe4fc4d26b75f2734953aabab9fdff83ce96"} Jul 20 02:45:39.185102 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:39.185067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:39.185102 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:39.185091 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:39.185370 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.185202 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:39.185567 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.185532 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:39.761259 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:39.761222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:39.761720 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:39.761306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:39.761720 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.761397 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:39.761720 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.761467 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.761447459 +0000 UTC m=+66.637249146 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:39.761720 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.761404 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:39.761720 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.761554 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret podName:2158b0e6-20c0-4fbf-9eb1-a45566f1b034 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.761538504 +0000 UTC m=+66.637340192 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret") pod "global-pull-secret-syncer-mm7wt" (UID: "2158b0e6-20c0-4fbf-9eb1-a45566f1b034") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:39.861631 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:39.861597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:39.861788 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.861766 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:39.861847 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.861793 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:39.861847 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.861802 2565 projected.go:196] Error preparing data for projected volume kube-api-access-8d4lb for pod openshift-network-diagnostics/network-check-target-v4smd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:39.861915 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:39.861852 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb podName:15f98c6a-9d1e-4d6e-8841-78f8c6eb871d nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.86183628 +0000 UTC m=+66.737637970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8d4lb" (UniqueName: "kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb") pod "network-check-target-v4smd" (UID: "15f98c6a-9d1e-4d6e-8841-78f8c6eb871d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:40.185586 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:40.185552 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:40.185757 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:40.185677 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:45:41.184814 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:41.184781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:41.185269 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:41.184920 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4smd" podUID="15f98c6a-9d1e-4d6e-8841-78f8c6eb871d" Jul 20 02:45:41.185269 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:41.185015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:41.185269 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:41.185137 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm7wt" podUID="2158b0e6-20c0-4fbf-9eb1-a45566f1b034" Jul 20 02:45:41.935115 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:41.935043 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-38.ec2.internal" event="NodeReady" Jul 20 02:45:41.935246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:41.935188 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 02:45:41.976902 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:41.976865 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xl7dx"] Jul 20 02:45:42.006596 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.006567 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tnzc7"] Jul 20 02:45:42.006744 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.006708 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.008898 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.008874 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 02:45:42.009017 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.008957 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 02:45:42.009089 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.009032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz45m\"" Jul 20 02:45:42.018853 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.018832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnzc7"] Jul 20 02:45:42.018954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.018868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xl7dx"] Jul 20 02:45:42.018994 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.018951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.021207 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.021175 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 02:45:42.021339 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.021254 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mhmn8\"" Jul 20 02:45:42.021409 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.021260 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 02:45:42.021783 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.021765 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 02:45:42.078720 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.078689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.078857 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.078754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-config-volume\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.078857 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.078777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czq85\" (UniqueName: \"kubernetes.io/projected/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-kube-api-access-czq85\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.078857 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.078806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-tmp-dir\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180062 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/993b4933-d42d-4203-b148-ca51bfb26d38-cert\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.180252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpn4\" (UniqueName: \"kubernetes.io/projected/993b4933-d42d-4203-b148-ca51bfb26d38-kube-api-access-rmpn4\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.180252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-config-volume\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180252 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:42.180229 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:42.180252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czq85\" (UniqueName: \"kubernetes.io/projected/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-kube-api-access-czq85\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180527 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-tmp-dir\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180527 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:42.180310 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:42.680278841 +0000 UTC m=+37.556080537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:45:42.180691 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-tmp-dir\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.180824 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.180799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-config-volume\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.184779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.184745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:45:42.186581 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.186524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vd4r\"" Jul 20 02:45:42.186581 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.186523 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 02:45:42.191271 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.191142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czq85\" (UniqueName: \"kubernetes.io/projected/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-kube-api-access-czq85\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.281132 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.281093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/993b4933-d42d-4203-b148-ca51bfb26d38-cert\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.281342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.281279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpn4\" (UniqueName: \"kubernetes.io/projected/993b4933-d42d-4203-b148-ca51bfb26d38-kube-api-access-rmpn4\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.283954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.283912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/993b4933-d42d-4203-b148-ca51bfb26d38-cert\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.296870 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.296849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpn4\" (UniqueName: \"kubernetes.io/projected/993b4933-d42d-4203-b148-ca51bfb26d38-kube-api-access-rmpn4\") pod \"ingress-canary-tnzc7\" (UID: \"993b4933-d42d-4203-b148-ca51bfb26d38\") " pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.328824 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.328795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnzc7" Jul 20 02:45:42.593839 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.593808 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnzc7"] Jul 20 02:45:42.656879 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:45:42.656843 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993b4933_d42d_4203_b148_ca51bfb26d38.slice/crio-f47bbcf67d6406a7a0429c95fa69b21c1a4f0135ee0e0e14bb664f492bfb3ac1 WatchSource:0}: Error finding container f47bbcf67d6406a7a0429c95fa69b21c1a4f0135ee0e0e14bb664f492bfb3ac1: Status 404 returned error can't find the container with id f47bbcf67d6406a7a0429c95fa69b21c1a4f0135ee0e0e14bb664f492bfb3ac1 Jul 20 02:45:42.685270 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:42.685243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:42.685730 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:42.685700 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:42.685899 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:42.685781 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:43.68576062 +0000 UTC m=+38.561562327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:45:43.185139 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.185105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:45:43.185332 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.185106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:45:43.187405 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.187256 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 02:45:43.187405 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.187281 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 02:45:43.187405 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.187331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jphph\"" Jul 20 02:45:43.188224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.187664 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 02:45:43.474501 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.474420 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="0f6a4c380a3a7bdcf2a3753cfd31aef2a5fc8ae6c4fbd4fe8fcd3ab8c480a15d" exitCode=0 Jul 20 02:45:43.474642 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.474495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"0f6a4c380a3a7bdcf2a3753cfd31aef2a5fc8ae6c4fbd4fe8fcd3ab8c480a15d"} Jul 20 02:45:43.475878 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.475853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnzc7" event={"ID":"993b4933-d42d-4203-b148-ca51bfb26d38","Type":"ContainerStarted","Data":"f47bbcf67d6406a7a0429c95fa69b21c1a4f0135ee0e0e14bb664f492bfb3ac1"} Jul 20 02:45:43.691972 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:43.691935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:43.692133 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:43.692048 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:43.692133 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:43.692103 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:45.692087457 +0000 UTC m=+40.567889146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:45:44.482603 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:44.482567 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d" containerID="aa48098c1fd490f053894c44c4db8621d2f8d96b3ec02cb0cb95c61833f68fba" exitCode=0 Jul 20 02:45:44.483072 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:44.482636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerDied","Data":"aa48098c1fd490f053894c44c4db8621d2f8d96b3ec02cb0cb95c61833f68fba"} Jul 20 02:45:45.486372 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:45.486097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnzc7" event={"ID":"993b4933-d42d-4203-b148-ca51bfb26d38","Type":"ContainerStarted","Data":"2405669dbe10ccf6843f0ac57aa82faa2033d20098437d81bd03e5286a0b63a0"} Jul 20 02:45:45.489335 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:45.489311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" event={"ID":"dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d","Type":"ContainerStarted","Data":"aa5b452517af9be6129368f13410ff48e38c31f1e7e328148142fb7e22222dd2"} Jul 20 02:45:45.500304 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:45.500249 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tnzc7" podStartSLOduration=2.314803278 podStartE2EDuration="4.500234535s" totalImagesPullingTime="2.185431257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:45.499338732 +0000 UTC m=+40.375140442" watchObservedRunningTime="2026-07-20 02:45:45.500234535 +0000 UTC m=+40.376036245" Jul 20 02:45:45.520526 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:45.520402 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lkr6c" podStartSLOduration=8.655464831 podStartE2EDuration="38.520384921s" totalImagesPullingTime="29.86492009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:45.518056502 +0000 UTC m=+40.393858214" watchObservedRunningTime="2026-07-20 02:45:45.520384921 +0000 UTC m=+40.396186636" Jul 20 02:45:45.707894 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:45.707860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:45.708056 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:45.708006 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:45.708093 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:45.708067 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:49.708050957 +0000 UTC m=+44.583852647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:45:49.733378 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:49.733335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:49.733839 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:49.733496 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:49.733839 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:49.733562 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:57.733546152 +0000 UTC m=+52.609347840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:45:57.787755 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:45:57.787707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:45:57.788220 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:57.787883 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:57.788220 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:45:57.787952 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:13.787933758 +0000 UTC m=+68.663735456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:46:07.463946 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:07.463919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzx5" Jul 20 02:46:11.724910 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.724876 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq"] Jul 20 02:46:11.727656 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.727640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.729710 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.729686 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 02:46:11.729710 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.729689 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 02:46:11.730422 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.730402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 02:46:11.730500 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.730420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 02:46:11.730500 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.730443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 02:46:11.730500 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.730443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 02:46:11.730638 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.730443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 02:46:11.736586 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.736567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq"] Jul 20 02:46:11.787459 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.787430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:46:11.787624 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.787478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:46:11.789278 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.789250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 02:46:11.789410 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.789250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 02:46:11.798606 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:11.798582 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:46:11.798705 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:11.798648 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:47:15.798632919 +0000 UTC m=+130.674434606 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : secret "metrics-daemon-secret" not found Jul 20 02:46:11.801926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.801905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2158b0e6-20c0-4fbf-9eb1-a45566f1b034-original-pull-secret\") pod \"global-pull-secret-syncer-mm7wt\" (UID: \"2158b0e6-20c0-4fbf-9eb1-a45566f1b034\") " pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:46:11.888425 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888425 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7752a6d-322d-44d7-9076-96f42e26455f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmc7\" (UniqueName: \"kubernetes.io/projected/f7752a6d-322d-44d7-9076-96f42e26455f-kube-api-access-gkmc7\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.888763 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.888659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:46:11.891031 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.891014 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 02:46:11.901229 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.901210 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 02:46:11.912962 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.912936 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4lb\" (UniqueName: \"kubernetes.io/projected/15f98c6a-9d1e-4d6e-8841-78f8c6eb871d-kube-api-access-8d4lb\") pod \"network-check-target-v4smd\" (UID: \"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d\") " pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:46:11.989146 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989146 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7752a6d-322d-44d7-9076-96f42e26455f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmc7\" (UniqueName: \"kubernetes.io/projected/f7752a6d-322d-44d7-9076-96f42e26455f-kube-api-access-gkmc7\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.989979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.989949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7752a6d-322d-44d7-9076-96f42e26455f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.991738 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.991711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.991845 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.991739 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-ca\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.991845 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.991822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.991955 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.991947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7752a6d-322d-44d7-9076-96f42e26455f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:11.995926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:11.995905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmc7\" (UniqueName: \"kubernetes.io/projected/f7752a6d-322d-44d7-9076-96f42e26455f-kube-api-access-gkmc7\") pod \"cluster-proxy-proxy-agent-d6869bc4-zn5lq\" (UID: \"f7752a6d-322d-44d7-9076-96f42e26455f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:12.006144 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.006126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jphph\"" Jul 20 02:46:12.010574 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.010550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm7wt" Jul 20 02:46:12.013799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.013785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:46:12.051779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.051739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:46:12.171368 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.169826 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v4smd"] Jul 20 02:46:12.173497 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:46:12.172925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f98c6a_9d1e_4d6e_8841_78f8c6eb871d.slice/crio-755f8c51726fccef6dcf5421ca2852cead3179c6451d8972ff972f5f9572e9c0 WatchSource:0}: Error finding container 755f8c51726fccef6dcf5421ca2852cead3179c6451d8972ff972f5f9572e9c0: Status 404 returned error can't find the container with id 755f8c51726fccef6dcf5421ca2852cead3179c6451d8972ff972f5f9572e9c0 Jul 20 02:46:12.188481 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.188451 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mm7wt"] Jul 20 02:46:12.191867 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:46:12.191838 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2158b0e6_20c0_4fbf_9eb1_a45566f1b034.slice/crio-1a7252ad3a6051cde4690bfa5d48f67b36ae8b2e8af09fa334495873033bc978 WatchSource:0}: Error finding container 1a7252ad3a6051cde4690bfa5d48f67b36ae8b2e8af09fa334495873033bc978: Status 404 returned error can't find the container with id 1a7252ad3a6051cde4690bfa5d48f67b36ae8b2e8af09fa334495873033bc978 Jul 20 02:46:12.218246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.218216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq"] Jul 20 02:46:12.221348 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:46:12.221285 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7752a6d_322d_44d7_9076_96f42e26455f.slice/crio-2ebc8feef615e80283e1ae812ca5d96586d902f529d3611f380c6223aeaa6828 WatchSource:0}: Error finding container 2ebc8feef615e80283e1ae812ca5d96586d902f529d3611f380c6223aeaa6828: Status 404 returned error can't find the container with id 2ebc8feef615e80283e1ae812ca5d96586d902f529d3611f380c6223aeaa6828 Jul 20 02:46:12.553525 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.553483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerStarted","Data":"2ebc8feef615e80283e1ae812ca5d96586d902f529d3611f380c6223aeaa6828"} Jul 20 02:46:12.554608 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.554582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v4smd" event={"ID":"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d","Type":"ContainerStarted","Data":"755f8c51726fccef6dcf5421ca2852cead3179c6451d8972ff972f5f9572e9c0"} Jul 20 02:46:12.555545 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:12.555523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mm7wt" event={"ID":"2158b0e6-20c0-4fbf-9eb1-a45566f1b034","Type":"ContainerStarted","Data":"1a7252ad3a6051cde4690bfa5d48f67b36ae8b2e8af09fa334495873033bc978"} Jul 20 02:46:13.804825 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:13.804785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:46:13.805316 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:13.804944 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:46:13.805316 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:13.805010 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:45.804993089 +0000 UTC m=+100.680794787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:46:18.574744 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:18.574696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerStarted","Data":"36d4863e8f3d7ec3a97d3c739bb20e21d9c88200894c4a4eb63d2828e17bb951"} Jul 20 02:46:18.576508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:18.576480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v4smd" event={"ID":"15f98c6a-9d1e-4d6e-8841-78f8c6eb871d","Type":"ContainerStarted","Data":"7c53e3c7e7f58b5adcded8123cea22a8bf0186c390838c59ef45056c8f4f94ca"} Jul 20 02:46:18.576679 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:18.576660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:46:18.591039 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:18.590463 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v4smd" podStartSLOduration=65.270812391 podStartE2EDuration="1m11.590439466s" totalImagesPullingTime="6.319627075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:18.589705318 +0000 UTC m=+73.465507030" watchObservedRunningTime="2026-07-20 02:46:18.590439466 +0000 UTC m=+73.466241176" Jul 20 02:46:19.581181 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:19.581144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mm7wt" event={"ID":"2158b0e6-20c0-4fbf-9eb1-a45566f1b034","Type":"ContainerStarted","Data":"a99063f8930a28c4c0ff637e0f8cbdc35185b5c0403e6dfcca39c523c04f6fec"} Jul 20 02:46:19.583238 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:19.583179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerStarted","Data":"22d2977541389f0a64ebae70b870c8f54198201675c47e9edcc25ccb5b0ae34d"} Jul 20 02:46:19.583374 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:19.583252 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerStarted","Data":"1c56d3c81a5f31a1aa034dc76ce292e7642065c2ddb302176c569257a82e4399"} Jul 20 02:46:19.609457 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:19.609408 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mm7wt" podStartSLOduration=66.297210029 podStartE2EDuration="1m12.609391244s" totalImagesPullingTime="6.312181215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.594314769 +0000 UTC m=+74.470116482" watchObservedRunningTime="2026-07-20 02:46:19.609391244 +0000 UTC m=+74.485192953" Jul 20 02:46:19.609672 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:19.609653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" podStartSLOduration=2.319330678 podStartE2EDuration="8.609644431s" totalImagesPullingTime="6.290313753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.609124049 +0000 UTC m=+74.484925759" watchObservedRunningTime="2026-07-20 02:46:19.609644431 +0000 UTC m=+74.485446140" Jul 20 02:46:45.834464 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:45.834403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:46:45.834874 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:45.834548 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:46:45.834874 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:46:45.834610 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:47:49.834596 +0000 UTC m=+164.710397688 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:46:49.585274 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:46:49.585241 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v4smd" Jul 20 02:47:15.851124 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:47:15.851082 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:47:15.851624 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:15.851229 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:47:15.851624 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:15.851322 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:17.851287878 +0000 UTC m=+252.727089566 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : secret "metrics-daemon-secret" not found Jul 20 02:47:45.019178 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:45.019113 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xl7dx" podUID="d64b4dc1-c25c-443f-a40a-55c43b1d19f8" Jul 20 02:47:45.200237 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:45.200192 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:47:45.822621 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:47:45.822592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:47:49.890741 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:47:49.890691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:47:49.891141 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:49.890839 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:47:49.891141 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:47:49.890910 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls podName:d64b4dc1-c25c-443f-a40a-55c43b1d19f8 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:51.890894343 +0000 UTC m=+286.766696034 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls") pod "dns-default-xl7dx" (UID: "d64b4dc1-c25c-443f-a40a-55c43b1d19f8") : secret "dns-default-metrics-tls" not found Jul 20 02:47:57.185811 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:47:57.185772 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:49:11.028408 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.028369 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4vsjm"] Jul 20 02:49:11.030406 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.030386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.033285 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.033264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dvbbm\"" Jul 20 02:49:11.034896 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.034875 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 20 02:49:11.035010 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.034923 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:49:11.035856 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.035831 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 20 02:49:11.036169 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.036154 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 20 02:49:11.039188 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.039171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 20 02:49:11.042013 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.041990 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4vsjm"] Jul 20 02:49:11.144622 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.144581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-config\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.144799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.144637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d74970a0-05c1-475e-b4bd-dc5a9e259f21-serving-cert\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.144799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.144744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-trusted-ca\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.144799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.144786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lfc\" (UniqueName: \"kubernetes.io/projected/d74970a0-05c1-475e-b4bd-dc5a9e259f21-kube-api-access-d6lfc\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.245671 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.245628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-trusted-ca\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.245846 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.245675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lfc\" (UniqueName: \"kubernetes.io/projected/d74970a0-05c1-475e-b4bd-dc5a9e259f21-kube-api-access-d6lfc\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.245846 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.245707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-config\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.245846 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.245768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d74970a0-05c1-475e-b4bd-dc5a9e259f21-serving-cert\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.246409 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.246390 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-config\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.246502 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.246465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d74970a0-05c1-475e-b4bd-dc5a9e259f21-trusted-ca\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.248260 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.248230 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d74970a0-05c1-475e-b4bd-dc5a9e259f21-serving-cert\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.253006 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.252976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lfc\" (UniqueName: \"kubernetes.io/projected/d74970a0-05c1-475e-b4bd-dc5a9e259f21-kube-api-access-d6lfc\") pod \"console-operator-565b7b569d-4vsjm\" (UID: \"d74970a0-05c1-475e-b4bd-dc5a9e259f21\") " pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.340300 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.340268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:11.457856 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:11.457826 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4vsjm"] Jul 20 02:49:11.461025 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:11.460995 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74970a0_05c1_475e_b4bd_dc5a9e259f21.slice/crio-4a9b04de2854041e9b33f5fe703af7663b4f1c8e51b514fcaaea26ba2c3bb4b2 WatchSource:0}: Error finding container 4a9b04de2854041e9b33f5fe703af7663b4f1c8e51b514fcaaea26ba2c3bb4b2: Status 404 returned error can't find the container with id 4a9b04de2854041e9b33f5fe703af7663b4f1c8e51b514fcaaea26ba2c3bb4b2 Jul 20 02:49:12.060141 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:12.060098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" event={"ID":"d74970a0-05c1-475e-b4bd-dc5a9e259f21","Type":"ContainerStarted","Data":"4a9b04de2854041e9b33f5fe703af7663b4f1c8e51b514fcaaea26ba2c3bb4b2"} Jul 20 02:49:14.066763 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.066733 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/0.log" Jul 20 02:49:14.067134 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.066772 2565 generic.go:356] "Generic (PLEG): container finished" podID="d74970a0-05c1-475e-b4bd-dc5a9e259f21" containerID="9bd6e97b86a887700ecabc0cf759756cdb9603b9f370eec9da89e616fb9189ad" exitCode=255 Jul 20 02:49:14.067134 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.066827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" event={"ID":"d74970a0-05c1-475e-b4bd-dc5a9e259f21","Type":"ContainerDied","Data":"9bd6e97b86a887700ecabc0cf759756cdb9603b9f370eec9da89e616fb9189ad"} Jul 20 02:49:14.067134 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.067037 2565 scope.go:117] "RemoveContainer" containerID="9bd6e97b86a887700ecabc0cf759756cdb9603b9f370eec9da89e616fb9189ad" Jul 20 02:49:14.601832 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.601799 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cwggl"] Jul 20 02:49:14.603750 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.603733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.605441 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.605419 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 02:49:14.605746 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.605717 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 02:49:14.605860 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.605776 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 02:49:14.605860 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.605802 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 02:49:14.606157 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.606139 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kg8wf\"" Jul 20 02:49:14.616623 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.616585 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cwggl"] Jul 20 02:49:14.671824 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.671791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/122e6085-1ccb-4bd9-bdcf-dd77366c6652-data-volume\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.671989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.671844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.671989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.671861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/122e6085-1ccb-4bd9-bdcf-dd77366c6652-crio-socket\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.671989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.671908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.671989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.671941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82kw2\" (UniqueName: \"kubernetes.io/projected/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-api-access-82kw2\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773128 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773128 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82kw2\" (UniqueName: \"kubernetes.io/projected/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-api-access-82kw2\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/122e6085-1ccb-4bd9-bdcf-dd77366c6652-data-volume\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:14.773243 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/122e6085-1ccb-4bd9-bdcf-dd77366c6652-crio-socket\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:14.773334 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:15.273312704 +0000 UTC m=+250.149114411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/122e6085-1ccb-4bd9-bdcf-dd77366c6652-crio-socket\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.773400 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:14.773356 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:14.773658 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:14.773423 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:15.273406039 +0000 UTC m=+250.149207747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : configmap "kube-rbac-proxy" not found Jul 20 02:49:14.773658 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.773649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/122e6085-1ccb-4bd9-bdcf-dd77366c6652-data-volume\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:14.782240 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:14.782218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82kw2\" (UniqueName: \"kubernetes.io/projected/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-api-access-82kw2\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:15.071042 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.070970 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:49:15.071406 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.071361 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/0.log" Jul 20 02:49:15.071406 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.071395 2565 generic.go:356] "Generic (PLEG): container finished" podID="d74970a0-05c1-475e-b4bd-dc5a9e259f21" containerID="b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034" exitCode=255 Jul 20 02:49:15.071527 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.071426 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" event={"ID":"d74970a0-05c1-475e-b4bd-dc5a9e259f21","Type":"ContainerDied","Data":"b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034"} Jul 20 02:49:15.071527 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.071485 2565 scope.go:117] "RemoveContainer" containerID="9bd6e97b86a887700ecabc0cf759756cdb9603b9f370eec9da89e616fb9189ad" Jul 20 02:49:15.071712 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.071696 2565 scope.go:117] "RemoveContainer" containerID="b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034" Jul 20 02:49:15.071884 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:15.071860 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4vsjm_openshift-console-operator(d74970a0-05c1-475e-b4bd-dc5a9e259f21)\"" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" podUID="d74970a0-05c1-475e-b4bd-dc5a9e259f21" Jul 20 02:49:15.277983 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.277943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:15.277983 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:15.277984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:15.278200 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:15.278059 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:15.278200 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:15.278122 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:16.278108382 +0000 UTC m=+251.153910073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : configmap "kube-rbac-proxy" not found Jul 20 02:49:15.278200 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:15.278169 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:15.278362 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:15.278228 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:16.278214658 +0000 UTC m=+251.154016346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.076342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:16.076312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:49:16.076829 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:16.076727 2565 scope.go:117] "RemoveContainer" containerID="b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034" Jul 20 02:49:16.076935 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:16.076913 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4vsjm_openshift-console-operator(d74970a0-05c1-475e-b4bd-dc5a9e259f21)\"" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" podUID="d74970a0-05c1-475e-b4bd-dc5a9e259f21" Jul 20 02:49:16.285623 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:16.285587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:16.285623 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:16.285624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:16.285903 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:16.285699 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:16.285903 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:16.285761 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:18.285746646 +0000 UTC m=+253.161548336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : configmap "kube-rbac-proxy" not found Jul 20 02:49:16.285903 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:16.285786 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.285903 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:16.285832 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:18.285818215 +0000 UTC m=+253.161619903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.382886 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:16.382818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gb6bp_b41a9a75-9ff7-4235-a372-66fc6b1dc25e/dns-node-resolver/0.log" Jul 20 02:49:17.783531 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:17.783503 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pl4t4_ea855527-0972-4f91-8790-fb5b6217bd38/node-ca/0.log" Jul 20 02:49:17.898194 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:17.898148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:49:17.898384 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:17.898306 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:49:17.898384 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:17.898374 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs podName:0cbf1d55-411c-48ce-b16f-33b6adf859e7 nodeName:}" failed. No retries permitted until 2026-07-20 02:51:19.898356531 +0000 UTC m=+374.774158222 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs") pod "network-metrics-daemon-2vxkh" (UID: "0cbf1d55-411c-48ce-b16f-33b6adf859e7") : secret "metrics-daemon-secret" not found Jul 20 02:49:18.301614 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:18.301573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:18.301614 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:18.301618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:18.301838 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:18.301704 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:18.301838 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:18.301780 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:22.301762268 +0000 UTC m=+257.177563984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : configmap "kube-rbac-proxy" not found Jul 20 02:49:18.301909 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:18.301862 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:18.301945 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:18.301929 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:22.301915489 +0000 UTC m=+257.177717192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:18.583872 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:18.583848 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnzc7_993b4933-d42d-4203-b148-ca51bfb26d38/serve-healthcheck-canary/0.log" Jul 20 02:49:21.340605 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:21.340565 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:21.340979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:21.340619 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:21.340979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:21.340910 2565 scope.go:117] "RemoveContainer" containerID="b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034" Jul 20 02:49:21.341068 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:21.341047 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4vsjm_openshift-console-operator(d74970a0-05c1-475e-b4bd-dc5a9e259f21)\"" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" podUID="d74970a0-05c1-475e-b4bd-dc5a9e259f21" Jul 20 02:49:22.333089 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:22.333051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:22.333322 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:22.333202 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:22.333322 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:22.333269 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:30.333253411 +0000 UTC m=+265.209055098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:22.333470 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:22.333449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:22.333566 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:22.333549 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:22.333622 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:22.333615 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm podName:122e6085-1ccb-4bd9-bdcf-dd77366c6652 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:30.333597383 +0000 UTC m=+265.209399087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cwggl" (UID: "122e6085-1ccb-4bd9-bdcf-dd77366c6652") : configmap "kube-rbac-proxy" not found Jul 20 02:49:30.393952 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.393861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:30.393952 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.393905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:30.394485 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.394460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/122e6085-1ccb-4bd9-bdcf-dd77366c6652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:30.396176 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.396158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/122e6085-1ccb-4bd9-bdcf-dd77366c6652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwggl\" (UID: \"122e6085-1ccb-4bd9-bdcf-dd77366c6652\") " pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:30.512066 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.512026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cwggl" Jul 20 02:49:30.636139 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:30.636115 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cwggl"] Jul 20 02:49:30.638625 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:30.638600 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122e6085_1ccb_4bd9_bdcf_dd77366c6652.slice/crio-571cfe1c5ba866976f0242fa3813b2f6d5a3c29f094d7a41598980f667fb57e9 WatchSource:0}: Error finding container 571cfe1c5ba866976f0242fa3813b2f6d5a3c29f094d7a41598980f667fb57e9: Status 404 returned error can't find the container with id 571cfe1c5ba866976f0242fa3813b2f6d5a3c29f094d7a41598980f667fb57e9 Jul 20 02:49:31.122005 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:31.121960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwggl" event={"ID":"122e6085-1ccb-4bd9-bdcf-dd77366c6652","Type":"ContainerStarted","Data":"571cfe1c5ba866976f0242fa3813b2f6d5a3c29f094d7a41598980f667fb57e9"} Jul 20 02:49:33.186087 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:33.186051 2565 scope.go:117] "RemoveContainer" containerID="b00afb47c94a464bfaf6a72f959470e59edc7df1f2f19fb223f57e19b7acc034" Jul 20 02:49:36.080535 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.080459 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7564878f47-6bcdn"] Jul 20 02:49:36.083526 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.083502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.086445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.086393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 02:49:36.086768 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.086747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 02:49:36.087094 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.086884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 02:49:36.087094 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.086966 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dwwjm\"" Jul 20 02:49:36.093446 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.093406 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 02:49:36.099010 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.098988 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7564878f47-6bcdn"] Jul 20 02:49:36.136902 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.136871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-tls\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137067 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.136927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-image-registry-private-configuration\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137067 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.136984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46026170-a1e1-4f36-97dc-f4c9fc2de40c-ca-trust-extracted\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137067 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.137025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxkg\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-kube-api-access-5sxkg\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137067 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.137051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-trusted-ca\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137372 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.137079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-bound-sa-token\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137372 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.137125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-installation-pull-secrets\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.137372 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.137152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-certificates\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.142520 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.142489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwggl" event={"ID":"122e6085-1ccb-4bd9-bdcf-dd77366c6652","Type":"ContainerStarted","Data":"1e27cb39ae7a097a51f41866da23702fc5eb3847faf185414cd0a6dd697dcb36"} Jul 20 02:49:36.144767 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.144742 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:49:36.144891 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.144795 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" event={"ID":"d74970a0-05c1-475e-b4bd-dc5a9e259f21","Type":"ContainerStarted","Data":"7ae4e3a5b52d0befff97af5547e79f55e82f37b58e9e320c482b7314cc5369e0"} Jul 20 02:49:36.145282 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.145265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:36.161508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.161457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" podStartSLOduration=23.116010198 podStartE2EDuration="25.161438099s" totalImagesPullingTime="2.045427901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:36.160522133 +0000 UTC m=+271.036323846" watchObservedRunningTime="2026-07-20 02:49:36.161438099 +0000 UTC m=+271.037239814" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-installation-pull-secrets\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-certificates\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-tls\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-image-registry-private-configuration\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46026170-a1e1-4f36-97dc-f4c9fc2de40c-ca-trust-extracted\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxkg\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-kube-api-access-5sxkg\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-trusted-ca\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.239316 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.238417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-bound-sa-token\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.240077 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.240022 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46026170-a1e1-4f36-97dc-f4c9fc2de40c-ca-trust-extracted\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.240576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.240546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-certificates\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.241675 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.241650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46026170-a1e1-4f36-97dc-f4c9fc2de40c-trusted-ca\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.241972 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.241929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-image-registry-private-configuration\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.242603 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.242582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46026170-a1e1-4f36-97dc-f4c9fc2de40c-installation-pull-secrets\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.242870 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.242853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-registry-tls\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.248887 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.248659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxkg\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-kube-api-access-5sxkg\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.248983 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.248929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46026170-a1e1-4f36-97dc-f4c9fc2de40c-bound-sa-token\") pod \"image-registry-7564878f47-6bcdn\" (UID: \"46026170-a1e1-4f36-97dc-f4c9fc2de40c\") " pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.253186 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.253038 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-4vsjm" Jul 20 02:49:36.397206 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.397106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:36.562736 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:36.562704 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7564878f47-6bcdn"] Jul 20 02:49:36.566029 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:36.565990 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46026170_a1e1_4f36_97dc_f4c9fc2de40c.slice/crio-c698a6ad0b2631e9fb234f18db275e5ee3412e8c385384e88df5b9968bc05a10 WatchSource:0}: Error finding container c698a6ad0b2631e9fb234f18db275e5ee3412e8c385384e88df5b9968bc05a10: Status 404 returned error can't find the container with id c698a6ad0b2631e9fb234f18db275e5ee3412e8c385384e88df5b9968bc05a10 Jul 20 02:49:37.149107 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:37.149067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" event={"ID":"46026170-a1e1-4f36-97dc-f4c9fc2de40c","Type":"ContainerStarted","Data":"3a7f4f2e09ea475594248034281d7b490cc06585c50cee7c79bc57cc593abe9b"} Jul 20 02:49:37.149556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:37.149127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" event={"ID":"46026170-a1e1-4f36-97dc-f4c9fc2de40c","Type":"ContainerStarted","Data":"c698a6ad0b2631e9fb234f18db275e5ee3412e8c385384e88df5b9968bc05a10"} Jul 20 02:49:37.149556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:37.149283 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:37.170976 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:37.167346 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" podStartSLOduration=1.16732482 podStartE2EDuration="1.16732482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:37.165391257 +0000 UTC m=+272.041192970" watchObservedRunningTime="2026-07-20 02:49:37.16732482 +0000 UTC m=+272.043126535" Jul 20 02:49:39.158003 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:39.157966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwggl" event={"ID":"122e6085-1ccb-4bd9-bdcf-dd77366c6652","Type":"ContainerStarted","Data":"fd615ae5c746193c79ecff6bda95906c05d37ea8809c8fdf88f15e8a2e758d38"} Jul 20 02:49:43.173736 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:43.173696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwggl" event={"ID":"122e6085-1ccb-4bd9-bdcf-dd77366c6652","Type":"ContainerStarted","Data":"da1937e66931fd6dc6f1c612fab8f6720829d58da2bed610a58e46a35935b455"} Jul 20 02:49:43.189619 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:43.189569 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cwggl" podStartSLOduration=17.451981793 podStartE2EDuration="29.189554234s" totalImagesPullingTime="11.737572441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:49:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:43.188567979 +0000 UTC m=+278.064369692" watchObservedRunningTime="2026-07-20 02:49:43.189554234 +0000 UTC m=+278.065355934" Jul 20 02:49:47.204577 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.204523 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l"] Jul 20 02:49:47.207826 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.207806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.209810 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.209788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 02:49:47.209909 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.209891 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 20 02:49:47.209970 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.209909 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 02:49:47.209970 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.209936 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 20 02:49:47.210387 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.210369 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 02:49:47.210554 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.210521 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2g5bg\"" Jul 20 02:49:47.217206 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.217186 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l"] Jul 20 02:49:47.220261 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.220243 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-c8fk6"] Jul 20 02:49:47.223616 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.223600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.225791 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.225772 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 20 02:49:47.225893 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.225864 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 20 02:49:47.225976 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.225961 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sx68t\"" Jul 20 02:49:47.226091 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.226077 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 20 02:49:47.235951 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.235931 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-c8fk6"] Jul 20 02:49:47.238817 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.238795 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gvqct"] Jul 20 02:49:47.241829 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.241814 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.243406 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.243390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z7hth\"" Jul 20 02:49:47.243583 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.243570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 02:49:47.243906 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.243893 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 02:49:47.243906 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.243898 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 02:49:47.318745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.318745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.318940 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.318940 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.318940 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-metrics-client-ca\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319035 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.319035 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.318996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.319035 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32784c61-0c3b-427c-9848-c2bfab6d7df4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.319122 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-wtmp\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319122 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwsh\" (UniqueName: \"kubernetes.io/projected/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-api-access-xkwsh\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.319122 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-sys\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319210 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.319210 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-textfile\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319210 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-tls\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvs9\" (UniqueName: \"kubernetes.io/projected/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-kube-api-access-dvvs9\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvfh\" (UniqueName: \"kubernetes.io/projected/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-kube-api-access-5fvfh\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.319338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-root\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.319459 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.319345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420152 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-sys\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420152 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-textfile\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-tls\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-sys\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvs9\" (UniqueName: \"kubernetes.io/projected/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-kube-api-access-dvvs9\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvfh\" (UniqueName: \"kubernetes.io/projected/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-kube-api-access-5fvfh\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-root\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-root\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:47.420555 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:47.420619 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls podName:32784c61-0c3b-427c-9848-c2bfab6d7df4 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:47.920599204 +0000 UTC m=+282.796400900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-c8fk6" (UID: "32784c61-0c3b-427c-9848-c2bfab6d7df4") : secret "kube-state-metrics-tls" not found Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420655 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-textfile\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-metrics-client-ca\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.420819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32784c61-0c3b-427c-9848-c2bfab6d7df4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-wtmp\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.420937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwsh\" (UniqueName: \"kubernetes.io/projected/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-api-access-xkwsh\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.421001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.421257 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.421019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.421612 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.421328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-wtmp\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.422046 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.422018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32784c61-0c3b-427c-9848-c2bfab6d7df4-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.422223 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.422186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.422223 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.422199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32784c61-0c3b-427c-9848-c2bfab6d7df4-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.422579 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.422552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-metrics-client-ca\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.423252 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.423226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-tls\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.423602 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.423575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.423770 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.423751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.424016 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.423996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.424180 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.424164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.429890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.429867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvs9\" (UniqueName: \"kubernetes.io/projected/c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2-kube-api-access-dvvs9\") pod \"node-exporter-gvqct\" (UID: \"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2\") " pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.430361 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.430341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvfh\" (UniqueName: \"kubernetes.io/projected/9324f46d-d7fd-4a95-9f49-b5dddca7cfde-kube-api-access-5fvfh\") pod \"openshift-state-metrics-79f6958cdf-cch6l\" (UID: \"9324f46d-d7fd-4a95-9f49-b5dddca7cfde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.430596 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.430576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwsh\" (UniqueName: \"kubernetes.io/projected/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-api-access-xkwsh\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.517718 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.517646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" Jul 20 02:49:47.549788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.549756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvqct" Jul 20 02:49:47.561059 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:47.561028 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b5cf75_76c5_4ea7_aa29_d328a32dd7a2.slice/crio-044477bd3fbcb188f2db5b7ca25a5275984d319831e504226a52605d33178c77 WatchSource:0}: Error finding container 044477bd3fbcb188f2db5b7ca25a5275984d319831e504226a52605d33178c77: Status 404 returned error can't find the container with id 044477bd3fbcb188f2db5b7ca25a5275984d319831e504226a52605d33178c77 Jul 20 02:49:47.643799 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.643767 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l"] Jul 20 02:49:47.646793 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:47.646763 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9324f46d_d7fd_4a95_9f49_b5dddca7cfde.slice/crio-38ce8371896134c6d903d8d97bab1f0487a9644ebb4fb838fb2bf159b7e4500d WatchSource:0}: Error finding container 38ce8371896134c6d903d8d97bab1f0487a9644ebb4fb838fb2bf159b7e4500d: Status 404 returned error can't find the container with id 38ce8371896134c6d903d8d97bab1f0487a9644ebb4fb838fb2bf159b7e4500d Jul 20 02:49:47.925097 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.925061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:47.927384 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:47.927360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32784c61-0c3b-427c-9848-c2bfab6d7df4-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-c8fk6\" (UID: \"32784c61-0c3b-427c-9848-c2bfab6d7df4\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:48.131886 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.131838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" Jul 20 02:49:48.190850 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.190817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" event={"ID":"9324f46d-d7fd-4a95-9f49-b5dddca7cfde","Type":"ContainerStarted","Data":"09f3b10a13e545e03c2350b55cea674c3f7fd82435b75eaa8bda36e154f47bc3"} Jul 20 02:49:48.190976 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.190890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" event={"ID":"9324f46d-d7fd-4a95-9f49-b5dddca7cfde","Type":"ContainerStarted","Data":"6d93e8039ee73e0867d1661a06de1516dafd8cb332fabd2d77d3012c17aada91"} Jul 20 02:49:48.190976 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.190921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" event={"ID":"9324f46d-d7fd-4a95-9f49-b5dddca7cfde","Type":"ContainerStarted","Data":"38ce8371896134c6d903d8d97bab1f0487a9644ebb4fb838fb2bf159b7e4500d"} Jul 20 02:49:48.194353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.194321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvqct" event={"ID":"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2","Type":"ContainerStarted","Data":"044477bd3fbcb188f2db5b7ca25a5275984d319831e504226a52605d33178c77"} Jul 20 02:49:48.266621 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.266587 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-c8fk6"] Jul 20 02:49:48.269718 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:48.269682 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32784c61_0c3b_427c_9848_c2bfab6d7df4.slice/crio-9383162828ac3ef7ca91aa3c09f2eefc8c456ee334c5daa4aa2de4e194e4114e WatchSource:0}: Error finding container 9383162828ac3ef7ca91aa3c09f2eefc8c456ee334c5daa4aa2de4e194e4114e: Status 404 returned error can't find the container with id 9383162828ac3ef7ca91aa3c09f2eefc8c456ee334c5daa4aa2de4e194e4114e Jul 20 02:49:48.823460 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:49:48.823414 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xl7dx" podUID="d64b4dc1-c25c-443f-a40a-55c43b1d19f8" Jul 20 02:49:48.940157 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.940132 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:49:48.944563 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.944545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:48.946602 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.946582 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 02:49:48.947393 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947375 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 02:49:48.947467 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 02:49:48.947467 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947414 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 02:49:48.947467 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947437 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 02:49:48.947609 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947542 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 02:49:48.947696 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5q4k\"" Jul 20 02:49:48.947753 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.947712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 02:49:48.952131 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.952022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 02:49:48.952705 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:48.952686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:49:49.036919 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.036886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.036919 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.036917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.037084 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.036940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.037084 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.037023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.037084 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.037061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjgr\" (UniqueName: \"kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.037187 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.037091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.037187 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.037130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138244 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138244 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjgr\" (UniqueName: \"kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.138488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.139019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.138990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.139345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.139321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.139474 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.139328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.139637 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.139607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.141505 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.141485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.141602 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.141508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.146235 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.146212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjgr\" (UniqueName: \"kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr\") pod \"console-59954c5f79-pzh94\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.200603 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.200563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" event={"ID":"9324f46d-d7fd-4a95-9f49-b5dddca7cfde","Type":"ContainerStarted","Data":"eddd22ee35e15d99a51ce6671fdbd03cc864fba30cb583b78cb75203ff49c9ac"} Jul 20 02:49:49.202009 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.201977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" event={"ID":"32784c61-0c3b-427c-9848-c2bfab6d7df4","Type":"ContainerStarted","Data":"9383162828ac3ef7ca91aa3c09f2eefc8c456ee334c5daa4aa2de4e194e4114e"} Jul 20 02:49:49.203566 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.203533 2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2" containerID="9ea49987fe7a8d42e45ee52c980a75d83765c90a19dd1a582a3c032e31d9a686" exitCode=0 Jul 20 02:49:49.203695 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.203613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvqct" event={"ID":"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2","Type":"ContainerDied","Data":"9ea49987fe7a8d42e45ee52c980a75d83765c90a19dd1a582a3c032e31d9a686"} Jul 20 02:49:49.203695 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.203659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:49:49.224406 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.224355 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cch6l" podStartSLOduration=1.069542047 podStartE2EDuration="2.22433752s" totalImagesPullingTime="1.154795473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:49.223324684 +0000 UTC m=+284.099126395" watchObservedRunningTime="2026-07-20 02:49:49.22433752 +0000 UTC m=+284.100139230" Jul 20 02:49:49.265751 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.265727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:49.413027 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:49.412969 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:49:49.415205 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:49.415175 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe226ecc_bb1c_4980_80f0_bfc3384c0143.slice/crio-8d32eff0f14087bf6aa12b566ce77c9d5d171acffcddfd547e708e580cac2e65 WatchSource:0}: Error finding container 8d32eff0f14087bf6aa12b566ce77c9d5d171acffcddfd547e708e580cac2e65: Status 404 returned error can't find the container with id 8d32eff0f14087bf6aa12b566ce77c9d5d171acffcddfd547e708e580cac2e65 Jul 20 02:49:50.211407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.211314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" event={"ID":"32784c61-0c3b-427c-9848-c2bfab6d7df4","Type":"ContainerStarted","Data":"90b50b03a51f5921292b8a036460456482a4f4a0cb480e1c57c1ee37d16c84ae"} Jul 20 02:49:50.211407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.211378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" event={"ID":"32784c61-0c3b-427c-9848-c2bfab6d7df4","Type":"ContainerStarted","Data":"e5fd7dd710351d0d54acb7d81fe9d3919037b70e5aaca1ccdf3eab6865635349"} Jul 20 02:49:50.211407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.211403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" event={"ID":"32784c61-0c3b-427c-9848-c2bfab6d7df4","Type":"ContainerStarted","Data":"62fa91cef2b29601746c3e33b9fe94273b426f170dccb67a3ef1246c619cff1a"} Jul 20 02:49:50.213074 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.213014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59954c5f79-pzh94" event={"ID":"fe226ecc-bb1c-4980-80f0-bfc3384c0143","Type":"ContainerStarted","Data":"8d32eff0f14087bf6aa12b566ce77c9d5d171acffcddfd547e708e580cac2e65"} Jul 20 02:49:50.217138 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.217055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvqct" event={"ID":"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2","Type":"ContainerStarted","Data":"379ec23c8d2e1d8ba86603551b1c8482625ab7a6351901175bd3a7a99e4f550f"} Jul 20 02:49:50.217138 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.217107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvqct" event={"ID":"c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2","Type":"ContainerStarted","Data":"72780c5519d51f16f3c153be62a9adb1c7a3f16ae8c25b9a7c7a5d18f6a5ce4a"} Jul 20 02:49:50.228957 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.228904 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-c8fk6" podStartSLOduration=1.642023322 podStartE2EDuration="3.22888795s" totalImagesPullingTime="1.586864628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:50.227812396 +0000 UTC m=+285.103614107" watchObservedRunningTime="2026-07-20 02:49:50.22888795 +0000 UTC m=+285.104689660" Jul 20 02:49:50.243340 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:50.243275 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gvqct" podStartSLOduration=2.114945189 podStartE2EDuration="3.243259411s" totalImagesPullingTime="1.0583712s" totalInitContainerRuntime="69.943022ms" isStatefulPod=true podCreationTimestamp="2026-07-20 02:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:50.242561043 +0000 UTC m=+285.118362758" watchObservedRunningTime="2026-07-20 02:49:50.243259411 +0000 UTC m=+285.119061121" Jul 20 02:49:51.566510 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.566475 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b596fcdc4-q4cqr"] Jul 20 02:49:51.569805 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.569786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.572103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572079 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vs5kp\"" Jul 20 02:49:51.572956 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572858 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 20 02:49:51.572956 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572896 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-da70vm9ttn02q\"" Jul 20 02:49:51.572956 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572902 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 02:49:51.573236 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 20 02:49:51.573236 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.572902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 20 02:49:51.577488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.577448 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b596fcdc4-q4cqr"] Jul 20 02:49:51.659573 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-audit-log\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-tls\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-metrics-server-audit-profiles\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxxx\" (UniqueName: \"kubernetes.io/projected/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-kube-api-access-bzxxx\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659745 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-client-certs\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659980 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.659980 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.659843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-client-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.760919 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.760879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-metrics-server-audit-profiles\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.760955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxxx\" (UniqueName: \"kubernetes.io/projected/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-kube-api-access-bzxxx\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-client-certs\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761074 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-client-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761335 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-audit-log\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761335 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-tls\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.761892 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761824 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.762008 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.761897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-audit-log\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.762071 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.762040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-metrics-server-audit-profiles\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.764079 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.764056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-client-ca-bundle\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.764198 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.764105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-client-certs\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.764544 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.764515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-secret-metrics-server-tls\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.769445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.769423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxxx\" (UniqueName: \"kubernetes.io/projected/b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b-kube-api-access-bzxxx\") pod \"metrics-server-b596fcdc4-q4cqr\" (UID: \"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b\") " pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.881805 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.881768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:49:51.962590 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.962552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:49:51.965199 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:51.965173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d64b4dc1-c25c-443f-a40a-55c43b1d19f8-metrics-tls\") pod \"dns-default-xl7dx\" (UID: \"d64b4dc1-c25c-443f-a40a-55c43b1d19f8\") " pod="openshift-dns/dns-default-xl7dx" Jul 20 02:49:52.205978 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:52.205954 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz45m\"" Jul 20 02:49:52.215532 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:52.215480 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:49:52.266956 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:52.266881 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b596fcdc4-q4cqr"] Jul 20 02:49:52.268920 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:52.268896 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b6a0ff_1c36_49fb_8ae7_f86ec51f488b.slice/crio-4750c00f28e96bd1a98a9bbfc3706cf4aabd229d0d04f7505e9c79af50f869c9 WatchSource:0}: Error finding container 4750c00f28e96bd1a98a9bbfc3706cf4aabd229d0d04f7505e9c79af50f869c9: Status 404 returned error can't find the container with id 4750c00f28e96bd1a98a9bbfc3706cf4aabd229d0d04f7505e9c79af50f869c9 Jul 20 02:49:52.354263 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:52.354212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xl7dx"] Jul 20 02:49:52.357627 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:49:52.357600 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64b4dc1_c25c_443f_a40a_55c43b1d19f8.slice/crio-da9e9b94304b46fadda9ec2fe3ddda3b072d69a0d096e03e1dc871cc4db9757a WatchSource:0}: Error finding container da9e9b94304b46fadda9ec2fe3ddda3b072d69a0d096e03e1dc871cc4db9757a: Status 404 returned error can't find the container with id da9e9b94304b46fadda9ec2fe3ddda3b072d69a0d096e03e1dc871cc4db9757a Jul 20 02:49:53.236801 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:53.236755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59954c5f79-pzh94" event={"ID":"fe226ecc-bb1c-4980-80f0-bfc3384c0143","Type":"ContainerStarted","Data":"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2"} Jul 20 02:49:53.238113 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:53.238086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" event={"ID":"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b","Type":"ContainerStarted","Data":"4750c00f28e96bd1a98a9bbfc3706cf4aabd229d0d04f7505e9c79af50f869c9"} Jul 20 02:49:53.239354 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:53.239324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl7dx" event={"ID":"d64b4dc1-c25c-443f-a40a-55c43b1d19f8","Type":"ContainerStarted","Data":"da9e9b94304b46fadda9ec2fe3ddda3b072d69a0d096e03e1dc871cc4db9757a"} Jul 20 02:49:53.252538 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:53.252491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59954c5f79-pzh94" podStartSLOduration=2.469269894 podStartE2EDuration="5.252476424s" totalImagesPullingTime="2.78320653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:53.251932724 +0000 UTC m=+288.127734435" watchObservedRunningTime="2026-07-20 02:49:53.252476424 +0000 UTC m=+288.128278128" Jul 20 02:49:55.248850 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.248815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" event={"ID":"b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b","Type":"ContainerStarted","Data":"33f404e9e08f9f2b7e62a377ec4ae18f8bb388abe319a80b44c07c4c9523ba3f"} Jul 20 02:49:55.250600 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.250573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl7dx" event={"ID":"d64b4dc1-c25c-443f-a40a-55c43b1d19f8","Type":"ContainerStarted","Data":"c52ef11c33e04ca48627efdac5e6c33a1d6e3e3c13dbf4206f12bb70c33744c7"} Jul 20 02:49:55.250733 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.250622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xl7dx" event={"ID":"d64b4dc1-c25c-443f-a40a-55c43b1d19f8","Type":"ContainerStarted","Data":"faa6aed9e196850ff11b62b70039d5a419e1a92ea9e1f8e4852b037607f6015f"} Jul 20 02:49:55.250733 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.250706 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:49:55.272374 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.272317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" podStartSLOduration=2.244894147 podStartE2EDuration="4.272283325s" totalImagesPullingTime="2.027389178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:55.271642145 +0000 UTC m=+290.147443859" watchObservedRunningTime="2026-07-20 02:49:55.272283325 +0000 UTC m=+290.148085034" Jul 20 02:49:55.289004 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:55.288955 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xl7dx" podStartSLOduration=252.345391595 podStartE2EDuration="4m14.288941977s" totalImagesPullingTime="1.943550382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:55.28757887 +0000 UTC m=+290.163380580" watchObservedRunningTime="2026-07-20 02:49:55.288941977 +0000 UTC m=+290.164743682" Jul 20 02:49:56.401622 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:56.401577 2565 patch_prober.go:28] interesting pod/image-registry-7564878f47-6bcdn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 02:49:56.401988 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:56.401639 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" podUID="46026170-a1e1-4f36-97dc-f4c9fc2de40c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 02:49:58.157435 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:58.157406 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7564878f47-6bcdn" Jul 20 02:49:59.266236 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:59.266198 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:59.266691 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:59.266260 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:49:59.271142 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:49:59.271120 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:50:00.186248 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:50:00.186193 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2vxkh" podUID="0cbf1d55-411c-48ce-b16f-33b6adf859e7" Jul 20 02:50:00.273593 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:00.273565 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:50:02.052986 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:02.052942 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" podUID="f7752a6d-322d-44d7-9076-96f42e26455f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 02:50:05.257726 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:05.257696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xl7dx" Jul 20 02:50:06.492545 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.492507 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:50:06.911601 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.911569 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:50:06.911771 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.911571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:50:06.924436 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.924414 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:50:06.924620 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.924599 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:50:06.929307 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:06.929275 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 02:50:11.185225 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:11.185185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:50:11.882083 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:11.882033 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:50:11.882258 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:11.882105 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:50:12.053777 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:12.053738 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" podUID="f7752a6d-322d-44d7-9076-96f42e26455f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 02:50:22.053613 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.053569 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" podUID="f7752a6d-322d-44d7-9076-96f42e26455f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 02:50:22.053979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.053660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" Jul 20 02:50:22.054230 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.054200 2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"22d2977541389f0a64ebae70b870c8f54198201675c47e9edcc25ccb5b0ae34d"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 20 02:50:22.054272 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.054254 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" podUID="f7752a6d-322d-44d7-9076-96f42e26455f" containerName="service-proxy" containerID="cri-o://22d2977541389f0a64ebae70b870c8f54198201675c47e9edcc25ccb5b0ae34d" gracePeriod=30 Jul 20 02:50:22.347732 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.347697 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7752a6d-322d-44d7-9076-96f42e26455f" containerID="22d2977541389f0a64ebae70b870c8f54198201675c47e9edcc25ccb5b0ae34d" exitCode=2 Jul 20 02:50:22.347904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.347746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerDied","Data":"22d2977541389f0a64ebae70b870c8f54198201675c47e9edcc25ccb5b0ae34d"} Jul 20 02:50:22.347904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:22.347785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d6869bc4-zn5lq" event={"ID":"f7752a6d-322d-44d7-9076-96f42e26455f","Type":"ContainerStarted","Data":"751e2069c7fbca7dffb1247c3a1353ff21f16d40b3b8cad27f76a00bf283d6d3"} Jul 20 02:50:31.519658 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.519599 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59954c5f79-pzh94" podUID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" containerName="console" containerID="cri-o://06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2" gracePeriod=15 Jul 20 02:50:31.752670 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.752649 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:50:31.811773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjgr\" (UniqueName: \"kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.811773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.811773 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811753 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.812019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811777 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.812019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811822 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.812019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.812019 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.811901 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config\") pod \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\" (UID: \"fe226ecc-bb1c-4980-80f0-bfc3384c0143\") " Jul 20 02:50:31.812236 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.812208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config" (OuterVolumeSpecName: "console-config") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:31.812339 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.812314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:31.812403 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.812351 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:31.812403 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.812351 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:31.814001 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.813966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:31.814103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.813972 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr" (OuterVolumeSpecName: "kube-api-access-jzjgr") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "kube-api-access-jzjgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:50:31.814103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.814012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe226ecc-bb1c-4980-80f0-bfc3384c0143" (UID: "fe226ecc-bb1c-4980-80f0-bfc3384c0143"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:31.887421 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.887388 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:50:31.891103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.891079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b596fcdc4-q4cqr" Jul 20 02:50:31.912620 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912583 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjgr\" (UniqueName: \"kubernetes.io/projected/fe226ecc-bb1c-4980-80f0-bfc3384c0143-kube-api-access-jzjgr\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912691 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-serving-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912712 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-config\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912727 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912741 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-trusted-ca-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912755 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe226ecc-bb1c-4980-80f0-bfc3384c0143-oauth-serving-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:31.912788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:31.912770 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe226ecc-bb1c-4980-80f0-bfc3384c0143-console-oauth-config\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:50:32.383725 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.383682 2565 generic.go:356] "Generic (PLEG): container finished" podID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" containerID="06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2" exitCode=0 Jul 20 02:50:32.383890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.383748 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59954c5f79-pzh94" Jul 20 02:50:32.383890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.383774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59954c5f79-pzh94" event={"ID":"fe226ecc-bb1c-4980-80f0-bfc3384c0143","Type":"ContainerDied","Data":"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2"} Jul 20 02:50:32.383890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.383829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59954c5f79-pzh94" event={"ID":"fe226ecc-bb1c-4980-80f0-bfc3384c0143","Type":"ContainerDied","Data":"8d32eff0f14087bf6aa12b566ce77c9d5d171acffcddfd547e708e580cac2e65"} Jul 20 02:50:32.383890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.383850 2565 scope.go:117] "RemoveContainer" containerID="06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2" Jul 20 02:50:32.393595 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.393572 2565 scope.go:117] "RemoveContainer" containerID="06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2" Jul 20 02:50:32.393966 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:50:32.393900 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2\": container with ID starting with 06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2 not found: ID does not exist" containerID="06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2" Jul 20 02:50:32.393966 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.393940 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2"} err="failed to get container status \"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2\": rpc error: code = NotFound desc = could not find container \"06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2\": container with ID starting with 06e8378547a3b7cc33bda9883ba7630a80d0d8d684bb60f1afe7f44322862de2 not found: ID does not exist" Jul 20 02:50:32.403464 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.403444 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:50:32.407229 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:32.407204 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59954c5f79-pzh94"] Jul 20 02:50:33.189938 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:50:33.189905 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" path="/var/lib/kubelet/pods/fe226ecc-bb1c-4980-80f0-bfc3384c0143/volumes" Jul 20 02:51:19.994238 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:19.994185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:51:19.996528 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:19.996506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbf1d55-411c-48ce-b16f-33b6adf859e7-metrics-certs\") pod \"network-metrics-daemon-2vxkh\" (UID: \"0cbf1d55-411c-48ce-b16f-33b6adf859e7\") " pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:51:20.187549 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:20.187515 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vd4r\"" Jul 20 02:51:20.196098 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:20.196075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxkh" Jul 20 02:51:20.316658 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:20.316635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vxkh"] Jul 20 02:51:20.319004 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:51:20.318971 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbf1d55_411c_48ce_b16f_33b6adf859e7.slice/crio-b310b02a820350b4d8343b5d4be8ef38cfd03906e4d14591ec8fc82e7de17a26 WatchSource:0}: Error finding container b310b02a820350b4d8343b5d4be8ef38cfd03906e4d14591ec8fc82e7de17a26: Status 404 returned error can't find the container with id b310b02a820350b4d8343b5d4be8ef38cfd03906e4d14591ec8fc82e7de17a26 Jul 20 02:51:20.320897 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:20.320876 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:51:20.546553 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:20.546470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxkh" event={"ID":"0cbf1d55-411c-48ce-b16f-33b6adf859e7","Type":"ContainerStarted","Data":"b310b02a820350b4d8343b5d4be8ef38cfd03906e4d14591ec8fc82e7de17a26"} Jul 20 02:51:21.551741 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:21.551696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxkh" event={"ID":"0cbf1d55-411c-48ce-b16f-33b6adf859e7","Type":"ContainerStarted","Data":"2fcf30c422f33d707e8e3f4ca1b6e2fdbb92fac0a092ad2f8af21e4cd9b8278f"} Jul 20 02:51:21.552141 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:21.551782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxkh" event={"ID":"0cbf1d55-411c-48ce-b16f-33b6adf859e7","Type":"ContainerStarted","Data":"087903704ff634225f814b2a94cebfec314daf7ebf4158f9f47bcb761d411a44"} Jul 20 02:51:21.569076 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:51:21.569029 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2vxkh" podStartSLOduration=373.778787971 podStartE2EDuration="6m14.569015542s" totalImagesPullingTime="790.227571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:51:21.568032952 +0000 UTC m=+376.443834662" watchObservedRunningTime="2026-07-20 02:51:21.569015542 +0000 UTC m=+376.444817252" Jul 20 02:53:29.512608 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.512569 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x"] Jul 20 02:53:29.513172 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.512853 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" containerName="console" Jul 20 02:53:29.513172 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.512865 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" containerName="console" Jul 20 02:53:29.513172 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.512930 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe226ecc-bb1c-4980-80f0-bfc3384c0143" containerName="console" Jul 20 02:53:29.515820 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.515802 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.517926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.517907 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 02:53:29.517926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.517922 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 02:53:29.518102 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.517953 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pb4sp\"" Jul 20 02:53:29.523371 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.523348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x"] Jul 20 02:53:29.582742 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.582710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.582884 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.582748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.582884 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.582776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpcp\" (UniqueName: \"kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.683519 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.683487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpcp\" (UniqueName: \"kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.683702 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.683545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.683702 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.683572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.683935 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.683914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.683996 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.683942 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.691268 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.691235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpcp\" (UniqueName: \"kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.825488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.825459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:29.947987 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.947956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x"] Jul 20 02:53:29.951202 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:53:29.951177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a39ac97_c398_427c_ac2b_ad976943d9f0.slice/crio-dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7 WatchSource:0}: Error finding container dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7: Status 404 returned error can't find the container with id dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7 Jul 20 02:53:29.992965 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:29.992928 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" event={"ID":"7a39ac97-c398-427c-ac2b-ad976943d9f0","Type":"ContainerStarted","Data":"dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7"} Jul 20 02:53:36.021412 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:36.021377 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerID="36a3f921b44656a208b547d93fd8a8d67e0966cefd3d81a9355876f21809f3e0" exitCode=0 Jul 20 02:53:36.021788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:36.021435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" event={"ID":"7a39ac97-c398-427c-ac2b-ad976943d9f0","Type":"ContainerDied","Data":"36a3f921b44656a208b547d93fd8a8d67e0966cefd3d81a9355876f21809f3e0"} Jul 20 02:53:39.034334 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:39.034281 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerID="c5f14acc853236dc2a74f3b4303e93a811a4e1c72f97dfcb637617dcf8e5b2b8" exitCode=0 Jul 20 02:53:39.034334 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:39.034324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" event={"ID":"7a39ac97-c398-427c-ac2b-ad976943d9f0","Type":"ContainerDied","Data":"c5f14acc853236dc2a74f3b4303e93a811a4e1c72f97dfcb637617dcf8e5b2b8"} Jul 20 02:53:47.068907 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:47.068869 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerID="5f7e91b7a11ada87c44ec65ae58fd3fa3532bdb72940ee78b2e0170482698bac" exitCode=0 Jul 20 02:53:47.069310 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:47.068941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" event={"ID":"7a39ac97-c398-427c-ac2b-ad976943d9f0","Type":"ContainerDied","Data":"5f7e91b7a11ada87c44ec65ae58fd3fa3532bdb72940ee78b2e0170482698bac"} Jul 20 02:53:48.200021 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.199997 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:48.339717 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.339681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util\") pod \"7a39ac97-c398-427c-ac2b-ad976943d9f0\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " Jul 20 02:53:48.339911 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.339773 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mpcp\" (UniqueName: \"kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp\") pod \"7a39ac97-c398-427c-ac2b-ad976943d9f0\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " Jul 20 02:53:48.339911 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.339801 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle\") pod \"7a39ac97-c398-427c-ac2b-ad976943d9f0\" (UID: \"7a39ac97-c398-427c-ac2b-ad976943d9f0\") " Jul 20 02:53:48.340486 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.340460 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle" (OuterVolumeSpecName: "bundle") pod "7a39ac97-c398-427c-ac2b-ad976943d9f0" (UID: "7a39ac97-c398-427c-ac2b-ad976943d9f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:53:48.342018 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.341996 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp" (OuterVolumeSpecName: "kube-api-access-7mpcp") pod "7a39ac97-c398-427c-ac2b-ad976943d9f0" (UID: "7a39ac97-c398-427c-ac2b-ad976943d9f0"). InnerVolumeSpecName "kube-api-access-7mpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:53:48.343779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.343757 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util" (OuterVolumeSpecName: "util") pod "7a39ac97-c398-427c-ac2b-ad976943d9f0" (UID: "7a39ac97-c398-427c-ac2b-ad976943d9f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:53:48.440875 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.440833 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:53:48.440875 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.440868 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a39ac97-c398-427c-ac2b-ad976943d9f0-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:53:48.440875 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:48.440881 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mpcp\" (UniqueName: \"kubernetes.io/projected/7a39ac97-c398-427c-ac2b-ad976943d9f0-kube-api-access-7mpcp\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:53:49.078206 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:49.078170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" event={"ID":"7a39ac97-c398-427c-ac2b-ad976943d9f0","Type":"ContainerDied","Data":"dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7"} Jul 20 02:53:49.078427 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:49.078221 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp44x" Jul 20 02:53:49.078427 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:49.078245 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dffec328328f268baf1e0a026ac6310b6ad9e48c8988cfdbf805a176f18391b7" Jul 20 02:53:56.923690 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923652 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k"] Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923923 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="extract" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923934 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="extract" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923946 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="util" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="util" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923958 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="pull" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.923964 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="pull" Jul 20 02:53:56.924159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.924018 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a39ac97-c398-427c-ac2b-ad976943d9f0" containerName="extract" Jul 20 02:53:56.972736 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.972699 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k"] Jul 20 02:53:56.972901 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.972839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:56.974813 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.974789 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 02:53:56.974813 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.974789 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 02:53:56.975346 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:56.975329 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pb4sp\"" Jul 20 02:53:57.006054 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.006028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.006207 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.006085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgcg\" (UniqueName: \"kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.006207 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.006108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.107182 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.107153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgcg\" (UniqueName: \"kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.107402 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.107188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.107402 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.107225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.107615 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.107596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.107682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.107639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.115574 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.115553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgcg\" (UniqueName: \"kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.281731 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.281618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:53:57.433158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:57.433132 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k"] Jul 20 02:53:57.435446 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:53:57.435416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e76f705_ccc6_48c0_a4c8_b02594770c6a.slice/crio-d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289 WatchSource:0}: Error finding container d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289: Status 404 returned error can't find the container with id d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289 Jul 20 02:53:58.110819 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:58.110783 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerID="f0ca5a61e969e905478bc65667cbf3e5fdbfa16c214d7915a9788e573d470d13" exitCode=0 Jul 20 02:53:58.111260 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:58.110866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerDied","Data":"f0ca5a61e969e905478bc65667cbf3e5fdbfa16c214d7915a9788e573d470d13"} Jul 20 02:53:58.111260 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:53:58.110912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerStarted","Data":"d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289"} Jul 20 02:54:00.121578 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:00.121547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerStarted","Data":"25435aa8a1083d40f712237ce4a244bd4f7a2dd79345dfb35bffb78dffcd3477"} Jul 20 02:54:01.127640 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:01.127582 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerID="25435aa8a1083d40f712237ce4a244bd4f7a2dd79345dfb35bffb78dffcd3477" exitCode=0 Jul 20 02:54:01.128103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:01.127663 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerDied","Data":"25435aa8a1083d40f712237ce4a244bd4f7a2dd79345dfb35bffb78dffcd3477"} Jul 20 02:54:02.133751 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:02.133712 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerID="8a453617aea79cdf9eab721f3808ce39f0a0d80c2fc5e3395698f7ea92f79cd7" exitCode=0 Jul 20 02:54:02.134127 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:02.133801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerDied","Data":"8a453617aea79cdf9eab721f3808ce39f0a0d80c2fc5e3395698f7ea92f79cd7"} Jul 20 02:54:03.263804 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.263778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:54:03.354710 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.354681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle\") pod \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " Jul 20 02:54:03.354886 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.354738 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util\") pod \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " Jul 20 02:54:03.354886 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.354778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dgcg\" (UniqueName: \"kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg\") pod \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\" (UID: \"1e76f705-ccc6-48c0-a4c8-b02594770c6a\") " Jul 20 02:54:03.355090 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.355068 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle" (OuterVolumeSpecName: "bundle") pod "1e76f705-ccc6-48c0-a4c8-b02594770c6a" (UID: "1e76f705-ccc6-48c0-a4c8-b02594770c6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:54:03.356902 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.356881 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg" (OuterVolumeSpecName: "kube-api-access-8dgcg") pod "1e76f705-ccc6-48c0-a4c8-b02594770c6a" (UID: "1e76f705-ccc6-48c0-a4c8-b02594770c6a"). InnerVolumeSpecName "kube-api-access-8dgcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:54:03.359140 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.359116 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util" (OuterVolumeSpecName: "util") pod "1e76f705-ccc6-48c0-a4c8-b02594770c6a" (UID: "1e76f705-ccc6-48c0-a4c8-b02594770c6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:54:03.455767 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.455667 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:03.455767 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.455708 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgcg\" (UniqueName: \"kubernetes.io/projected/1e76f705-ccc6-48c0-a4c8-b02594770c6a-kube-api-access-8dgcg\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:03.455767 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:03.455719 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e76f705-ccc6-48c0-a4c8-b02594770c6a-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:04.142779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:04.142741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" event={"ID":"1e76f705-ccc6-48c0-a4c8-b02594770c6a","Type":"ContainerDied","Data":"d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289"} Jul 20 02:54:04.142779 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:04.142778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdb6k" Jul 20 02:54:04.142977 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:04.142792 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94e74dc4a99c4c6290bd1b05f16f77b8e96844deaf10d6f0422af02602bf289" Jul 20 02:54:09.724508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724477 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7"] Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724722 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="extract" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724732 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="extract" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724741 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="util" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724747 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="util" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724758 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="pull" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724763 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="pull" Jul 20 02:54:09.724904 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.724815 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e76f705-ccc6-48c0-a4c8-b02594770c6a" containerName="extract" Jul 20 02:54:09.728489 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.728474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.730651 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.730629 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:54:09.731175 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.731154 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 02:54:09.731283 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.731185 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kms2c\"" Jul 20 02:54:09.734747 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.734720 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7"] Jul 20 02:54:09.811608 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.811575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b8853a-131b-41f6-959a-660ad5b6812a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.811789 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.811640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6988\" (UniqueName: \"kubernetes.io/projected/50b8853a-131b-41f6-959a-660ad5b6812a-kube-api-access-k6988\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.912360 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.912321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6988\" (UniqueName: \"kubernetes.io/projected/50b8853a-131b-41f6-959a-660ad5b6812a-kube-api-access-k6988\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.912531 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.912370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b8853a-131b-41f6-959a-660ad5b6812a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.912789 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.912771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b8853a-131b-41f6-959a-660ad5b6812a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:09.921189 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:09.921163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6988\" (UniqueName: \"kubernetes.io/projected/50b8853a-131b-41f6-959a-660ad5b6812a-kube-api-access-k6988\") pod \"openshift-lws-operator-fd8ccff4c-vp4q7\" (UID: \"50b8853a-131b-41f6-959a-660ad5b6812a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:10.038262 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:10.038173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" Jul 20 02:54:10.165420 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:10.165390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7"] Jul 20 02:54:10.168561 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:54:10.168536 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50b8853a_131b_41f6_959a_660ad5b6812a.slice/crio-ee0a2226e2f0d235504a96f3cc614bb6246e266bfbd5395aff1ed84d106727d9 WatchSource:0}: Error finding container ee0a2226e2f0d235504a96f3cc614bb6246e266bfbd5395aff1ed84d106727d9: Status 404 returned error can't find the container with id ee0a2226e2f0d235504a96f3cc614bb6246e266bfbd5395aff1ed84d106727d9 Jul 20 02:54:11.169708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:11.169674 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" event={"ID":"50b8853a-131b-41f6-959a-660ad5b6812a","Type":"ContainerStarted","Data":"ee0a2226e2f0d235504a96f3cc614bb6246e266bfbd5395aff1ed84d106727d9"} Jul 20 02:54:19.201982 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:19.201945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" event={"ID":"50b8853a-131b-41f6-959a-660ad5b6812a","Type":"ContainerStarted","Data":"2de9ee18eb5a2b54821453c5e9c9bed8c88f15f2878dde53fcaac9dc9c773c23"} Jul 20 02:54:19.217029 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:19.216977 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vp4q7" podStartSLOduration=1.909828084 podStartE2EDuration="10.216961636s" totalImagesPullingTime="8.307133552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:19.215826833 +0000 UTC m=+554.091628544" watchObservedRunningTime="2026-07-20 02:54:19.216961636 +0000 UTC m=+554.092763345" Jul 20 02:54:22.354526 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.354487 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm"] Jul 20 02:54:22.357816 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.357800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.360082 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.360060 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pb4sp\"" Jul 20 02:54:22.360207 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.360086 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 02:54:22.360668 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.360653 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 02:54:22.365141 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.365113 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm"] Jul 20 02:54:22.417946 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.417910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.418117 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.417962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chldw\" (UniqueName: \"kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.418117 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.418063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.518893 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.518853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.519027 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.518926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.519027 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.518975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chldw\" (UniqueName: \"kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.519378 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.519341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.519437 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.519352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.526687 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.526661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chldw\" (UniqueName: \"kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.667414 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.667327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:22.792734 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:22.792708 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm"] Jul 20 02:54:22.794764 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:54:22.794735 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772367fe_2e70_4176_8af2_9122495e7ad5.slice/crio-8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0 WatchSource:0}: Error finding container 8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0: Status 404 returned error can't find the container with id 8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0 Jul 20 02:54:23.217601 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:23.217566 2565 generic.go:356] "Generic (PLEG): container finished" podID="772367fe-2e70-4176-8af2-9122495e7ad5" containerID="77f540e34f3960eff32de9d194036469e313a244e3fff97c3714d308e46cf420" exitCode=0 Jul 20 02:54:23.217765 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:23.217608 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" event={"ID":"772367fe-2e70-4176-8af2-9122495e7ad5","Type":"ContainerDied","Data":"77f540e34f3960eff32de9d194036469e313a244e3fff97c3714d308e46cf420"} Jul 20 02:54:23.217765 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:23.217651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" event={"ID":"772367fe-2e70-4176-8af2-9122495e7ad5","Type":"ContainerStarted","Data":"8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0"} Jul 20 02:54:24.223261 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:24.223173 2565 generic.go:356] "Generic (PLEG): container finished" podID="772367fe-2e70-4176-8af2-9122495e7ad5" containerID="932087d468f61b3b46f8b6aa30a4e66445ffab8439b19e5b434efa85aa62bcc9" exitCode=0 Jul 20 02:54:24.223261 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:24.223226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" event={"ID":"772367fe-2e70-4176-8af2-9122495e7ad5","Type":"ContainerDied","Data":"932087d468f61b3b46f8b6aa30a4e66445ffab8439b19e5b434efa85aa62bcc9"} Jul 20 02:54:25.228384 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:25.228346 2565 generic.go:356] "Generic (PLEG): container finished" podID="772367fe-2e70-4176-8af2-9122495e7ad5" containerID="d82b893acb32c091c07f1e44039dc7e2ef0e603a6e658d1f1bb2188757951dad" exitCode=0 Jul 20 02:54:25.228790 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:25.228427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" event={"ID":"772367fe-2e70-4176-8af2-9122495e7ad5","Type":"ContainerDied","Data":"d82b893acb32c091c07f1e44039dc7e2ef0e603a6e658d1f1bb2188757951dad"} Jul 20 02:54:26.357216 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.357189 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:26.449017 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.448980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle\") pod \"772367fe-2e70-4176-8af2-9122495e7ad5\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " Jul 20 02:54:26.449186 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.449035 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chldw\" (UniqueName: \"kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw\") pod \"772367fe-2e70-4176-8af2-9122495e7ad5\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " Jul 20 02:54:26.449186 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.449058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util\") pod \"772367fe-2e70-4176-8af2-9122495e7ad5\" (UID: \"772367fe-2e70-4176-8af2-9122495e7ad5\") " Jul 20 02:54:26.449822 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.449782 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle" (OuterVolumeSpecName: "bundle") pod "772367fe-2e70-4176-8af2-9122495e7ad5" (UID: "772367fe-2e70-4176-8af2-9122495e7ad5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:54:26.451126 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.451097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw" (OuterVolumeSpecName: "kube-api-access-chldw") pod "772367fe-2e70-4176-8af2-9122495e7ad5" (UID: "772367fe-2e70-4176-8af2-9122495e7ad5"). InnerVolumeSpecName "kube-api-access-chldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:54:26.454542 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.454518 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util" (OuterVolumeSpecName: "util") pod "772367fe-2e70-4176-8af2-9122495e7ad5" (UID: "772367fe-2e70-4176-8af2-9122495e7ad5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:54:26.550349 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.550231 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:26.550349 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.550264 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chldw\" (UniqueName: \"kubernetes.io/projected/772367fe-2e70-4176-8af2-9122495e7ad5-kube-api-access-chldw\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:26.550349 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:26.550274 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/772367fe-2e70-4176-8af2-9122495e7ad5-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:54:27.237737 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:27.237700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" event={"ID":"772367fe-2e70-4176-8af2-9122495e7ad5","Type":"ContainerDied","Data":"8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0"} Jul 20 02:54:27.237897 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:27.237758 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d77987cf7618d78bf4b71483ae59cfe37703c929f5a57006b794cfaa7de4ab0" Jul 20 02:54:27.237897 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:27.237710 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxgtm" Jul 20 02:54:34.848611 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb"] Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848847 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="util" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848857 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="util" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848864 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="pull" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848870 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="pull" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848880 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="extract" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848885 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="extract" Jul 20 02:54:34.849069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.848934 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="772367fe-2e70-4176-8af2-9122495e7ad5" containerName="extract" Jul 20 02:54:34.854959 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.854940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:34.856839 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.856819 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 02:54:34.861726 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.861705 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lp5sr\"" Jul 20 02:54:34.861809 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.861779 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 02:54:34.862394 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.862379 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 02:54:34.862478 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.862429 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 02:54:34.884775 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:34.884748 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb"] Jul 20 02:54:35.015989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.015955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.015989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.015997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.016246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.016019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.016246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.016078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.016246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.016175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sw6\" (UniqueName: \"kubernetes.io/projected/11461e94-3b0d-4e4e-9137-995511b4ceb0-kube-api-access-s7sw6\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117373 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117373 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117373 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117650 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117650 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sw6\" (UniqueName: \"kubernetes.io/projected/11461e94-3b0d-4e4e-9137-995511b4ceb0-kube-api-access-s7sw6\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.117737 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.117718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.119698 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.119672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11461e94-3b0d-4e4e-9137-995511b4ceb0-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.119841 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.119816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.119941 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.119928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11461e94-3b0d-4e4e-9137-995511b4ceb0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.125552 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.125528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sw6\" (UniqueName: \"kubernetes.io/projected/11461e94-3b0d-4e4e-9137-995511b4ceb0-kube-api-access-s7sw6\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb\" (UID: \"11461e94-3b0d-4e4e-9137-995511b4ceb0\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.164378 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.164341 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:35.312180 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:35.312154 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb"] Jul 20 02:54:35.314822 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:54:35.314796 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11461e94_3b0d_4e4e_9137_995511b4ceb0.slice/crio-45546b7bc9db0506ca3161089a36775dc650c053a83dd17012c21433057d17e2 WatchSource:0}: Error finding container 45546b7bc9db0506ca3161089a36775dc650c053a83dd17012c21433057d17e2: Status 404 returned error can't find the container with id 45546b7bc9db0506ca3161089a36775dc650c053a83dd17012c21433057d17e2 Jul 20 02:54:36.277386 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:36.277351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" event={"ID":"11461e94-3b0d-4e4e-9137-995511b4ceb0","Type":"ContainerStarted","Data":"45546b7bc9db0506ca3161089a36775dc650c053a83dd17012c21433057d17e2"} Jul 20 02:54:39.290242 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:39.290204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" event={"ID":"11461e94-3b0d-4e4e-9137-995511b4ceb0","Type":"ContainerStarted","Data":"519fad2d5095eefe8ef180e49f6df048d2801593eb002543702ae112ac7380d4"} Jul 20 02:54:40.295704 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:40.295662 2565 generic.go:356] "Generic (PLEG): container finished" podID="11461e94-3b0d-4e4e-9137-995511b4ceb0" containerID="519fad2d5095eefe8ef180e49f6df048d2801593eb002543702ae112ac7380d4" exitCode=0 Jul 20 02:54:40.295704 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:40.295707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" event={"ID":"11461e94-3b0d-4e4e-9137-995511b4ceb0","Type":"ContainerDied","Data":"519fad2d5095eefe8ef180e49f6df048d2801593eb002543702ae112ac7380d4"} Jul 20 02:54:41.302075 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:41.302035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" event={"ID":"11461e94-3b0d-4e4e-9137-995511b4ceb0","Type":"ContainerStarted","Data":"d08ecff28c4b2cbba47920a17b10a42c4a1f5192763556c23973cb9840718403"} Jul 20 02:54:41.302559 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:41.302157 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:41.322541 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:41.322486 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" podStartSLOduration=3.059602657 podStartE2EDuration="7.322470518s" totalImagesPullingTime="3.202088283s" totalInitContainerRuntime="1.060779578s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:41.320620651 +0000 UTC m=+576.196422362" watchObservedRunningTime="2026-07-20 02:54:41.322470518 +0000 UTC m=+576.198272228" Jul 20 02:54:52.308530 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:52.308498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb" Jul 20 02:54:54.359705 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.359666 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d88568984-wwmc2"] Jul 20 02:54:54.369882 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.369858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.372721 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.372512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t7lqz\"" Jul 20 02:54:54.377859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.372936 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 02:54:54.377859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.373045 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 02:54:54.377859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.373228 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 02:54:54.377859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.373280 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 02:54:54.378158 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.378097 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d88568984-wwmc2"] Jul 20 02:54:54.473576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.473544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26fw\" (UniqueName: \"kubernetes.io/projected/98682004-cb14-4a45-afea-0f41e3695832-kube-api-access-h26fw\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.473739 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.473596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98682004-cb14-4a45-afea-0f41e3695832-tls-certs\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.473739 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.473633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98682004-cb14-4a45-afea-0f41e3695832-tmp\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.575000 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.574960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26fw\" (UniqueName: \"kubernetes.io/projected/98682004-cb14-4a45-afea-0f41e3695832-kube-api-access-h26fw\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.575156 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.575016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98682004-cb14-4a45-afea-0f41e3695832-tls-certs\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.575156 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.575034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98682004-cb14-4a45-afea-0f41e3695832-tmp\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.577376 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.577349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98682004-cb14-4a45-afea-0f41e3695832-tmp\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.577496 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.577472 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98682004-cb14-4a45-afea-0f41e3695832-tls-certs\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.584021 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.583999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26fw\" (UniqueName: \"kubernetes.io/projected/98682004-cb14-4a45-afea-0f41e3695832-kube-api-access-h26fw\") pod \"kube-auth-proxy-6d88568984-wwmc2\" (UID: \"98682004-cb14-4a45-afea-0f41e3695832\") " pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.684782 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.684701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" Jul 20 02:54:54.814576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:54.814545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d88568984-wwmc2"] Jul 20 02:54:54.816973 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:54:54.816946 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98682004_cb14_4a45_afea_0f41e3695832.slice/crio-b90a2c4255cbf40649f94cc30c2ecfbd07321f170e93e573126609f6033f28da WatchSource:0}: Error finding container b90a2c4255cbf40649f94cc30c2ecfbd07321f170e93e573126609f6033f28da: Status 404 returned error can't find the container with id b90a2c4255cbf40649f94cc30c2ecfbd07321f170e93e573126609f6033f28da Jul 20 02:54:55.361070 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.361034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" event={"ID":"98682004-cb14-4a45-afea-0f41e3695832","Type":"ContainerStarted","Data":"b90a2c4255cbf40649f94cc30c2ecfbd07321f170e93e573126609f6033f28da"} Jul 20 02:54:55.681096 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.681014 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:54:55.684447 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.684421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.686713 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.686603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 02:54:55.686713 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.686622 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 02:54:55.686713 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.686601 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 02:54:55.686950 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.686933 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 02:54:55.687006 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.686989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z4wqv\"" Jul 20 02:54:55.698425 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.698402 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:54:55.782964 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.782932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783142 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.782980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783142 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.783026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783142 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.783096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmth\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783284 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.783157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783284 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.783187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.783284 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.783215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884621 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmth\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.884815 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.885052 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.884831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.885921 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.885861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.887558 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.887535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.887736 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.887709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.887847 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.887825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.887952 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.887935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.892240 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.892217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmth\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.892384 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.892365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token\") pod \"istiod-openshift-gateway-7574459656-tn6wt\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:55.997508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:55.996694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:54:56.170945 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:56.170904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:54:56.172817 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:54:56.172782 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4514f897_672c_4dc0_8172_fafa747b9395.slice/crio-7ba9436258003f742a44b150ba94d9e5c566a9a52bc7753c9eafb7d38dbcb272 WatchSource:0}: Error finding container 7ba9436258003f742a44b150ba94d9e5c566a9a52bc7753c9eafb7d38dbcb272: Status 404 returned error can't find the container with id 7ba9436258003f742a44b150ba94d9e5c566a9a52bc7753c9eafb7d38dbcb272 Jul 20 02:54:56.367416 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:56.367371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" event={"ID":"4514f897-672c-4dc0-8172-fafa747b9395","Type":"ContainerStarted","Data":"7ba9436258003f742a44b150ba94d9e5c566a9a52bc7753c9eafb7d38dbcb272"} Jul 20 02:54:57.375387 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:57.375349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" event={"ID":"98682004-cb14-4a45-afea-0f41e3695832","Type":"ContainerStarted","Data":"3678e557241accef2ada96a8a4467a514620ce496a213145f88b9503957bac89"} Jul 20 02:54:57.391783 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:54:57.391731 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d88568984-wwmc2" podStartSLOduration=1.015528326 podStartE2EDuration="3.391714383s" totalImagesPullingTime="2.376186057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:57.389861892 +0000 UTC m=+592.265663614" watchObservedRunningTime="2026-07-20 02:54:57.391714383 +0000 UTC m=+592.267516093" Jul 20 02:55:00.882300 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:00.882250 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 02:55:00.882637 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:00.882341 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 02:55:01.393793 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:01.393758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" event={"ID":"4514f897-672c-4dc0-8172-fafa747b9395","Type":"ContainerStarted","Data":"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd"} Jul 20 02:55:01.393989 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:01.393958 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:55:01.395766 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:01.395740 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-tn6wt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 02:55:01.395889 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:01.395793 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 02:55:01.416893 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:01.416845 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" podStartSLOduration=1.710030722 podStartE2EDuration="6.416829833s" totalImagesPullingTime="4.706799111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:01.414864033 +0000 UTC m=+596.290665745" watchObservedRunningTime="2026-07-20 02:55:01.416829833 +0000 UTC m=+596.292631543" Jul 20 02:55:02.398564 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:02.398531 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-tn6wt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 02:55:02.398949 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:02.398602 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 02:55:04.328241 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.328203 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv"] Jul 20 02:55:04.331386 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.331367 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.334162 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.334140 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 20 02:55:04.334246 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.334142 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-69r4r\"" Jul 20 02:55:04.337340 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.337318 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8"] Jul 20 02:55:04.340567 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.340550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.346159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.345376 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 20 02:55:04.346159 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.346054 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2xcjh\"" Jul 20 02:55:04.349794 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.349754 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv"] Jul 20 02:55:04.361432 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.361403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.361582 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.361477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5k4\" (UniqueName: \"kubernetes.io/projected/1b6c66e2-5522-4894-a420-3e0381a84cbf-kube-api-access-fq5k4\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.361963 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.361942 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8"] Jul 20 02:55:04.462522 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.462485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.462715 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.462559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5k4\" (UniqueName: \"kubernetes.io/projected/1b6c66e2-5522-4894-a420-3e0381a84cbf-kube-api-access-fq5k4\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.462715 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.462648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbljf\" (UniqueName: \"kubernetes.io/projected/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-kube-api-access-xbljf\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.462715 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.462667 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 20 02:55:04.462887 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.462778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.462950 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.462914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert podName:1b6c66e2-5522-4894-a420-3e0381a84cbf nodeName:}" failed. No retries permitted until 2026-07-20 02:55:04.962886748 +0000 UTC m=+599.838688439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert") pod "kserve-controller-manager-d6f4b65b8-pb9bv" (UID: "1b6c66e2-5522-4894-a420-3e0381a84cbf") : secret "kserve-webhook-server-cert" not found Jul 20 02:55:04.481340 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.481276 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5k4\" (UniqueName: \"kubernetes.io/projected/1b6c66e2-5522-4894-a420-3e0381a84cbf-kube-api-access-fq5k4\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.563193 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.563156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbljf\" (UniqueName: \"kubernetes.io/projected/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-kube-api-access-xbljf\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.563399 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.563201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.563399 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.563345 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 20 02:55:04.563399 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.563400 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert podName:abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d nodeName:}" failed. No retries permitted until 2026-07-20 02:55:05.063385162 +0000 UTC m=+599.939186850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert") pod "llmisvc-controller-manager-777c6bd7d5-5msd8" (UID: "abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d") : secret "llmisvc-webhook-server-cert" not found Jul 20 02:55:04.578080 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.578047 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbljf\" (UniqueName: \"kubernetes.io/projected/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-kube-api-access-xbljf\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:04.966379 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:04.966344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:04.966564 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.966475 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 20 02:55:04.966564 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:04.966537 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert podName:1b6c66e2-5522-4894-a420-3e0381a84cbf nodeName:}" failed. No retries permitted until 2026-07-20 02:55:05.966522332 +0000 UTC m=+600.842324030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert") pod "kserve-controller-manager-d6f4b65b8-pb9bv" (UID: "1b6c66e2-5522-4894-a420-3e0381a84cbf") : secret "kserve-webhook-server-cert" not found Jul 20 02:55:05.066810 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.066777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:05.069627 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.069600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5msd8\" (UID: \"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:05.253522 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.253437 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:05.398993 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.398963 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:55:05.404327 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.404261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8"] Jul 20 02:55:05.406467 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:55:05.406432 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podabfb8c3c_fd68_4a72_ae16_5f30b8e09d2d.slice/crio-597197ce881d270d3da18e427384320eb5f03709496cfd52f6bff3b2d7b93388 WatchSource:0}: Error finding container 597197ce881d270d3da18e427384320eb5f03709496cfd52f6bff3b2d7b93388: Status 404 returned error can't find the container with id 597197ce881d270d3da18e427384320eb5f03709496cfd52f6bff3b2d7b93388 Jul 20 02:55:05.974686 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.974637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:05.977671 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:05.977645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6c66e2-5522-4894-a420-3e0381a84cbf-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb9bv\" (UID: \"1b6c66e2-5522-4894-a420-3e0381a84cbf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:06.142506 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.142467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:06.416034 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.415997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" event={"ID":"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d","Type":"ContainerStarted","Data":"597197ce881d270d3da18e427384320eb5f03709496cfd52f6bff3b2d7b93388"} Jul 20 02:55:06.648379 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.648140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv"] Jul 20 02:55:06.690465 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:55:06.690393 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6c66e2_5522_4894_a420_3e0381a84cbf.slice/crio-2cc8d68839ea61acdbaa34a2782211fadaeacb3395e33538d7c701c8ac8e492c WatchSource:0}: Error finding container 2cc8d68839ea61acdbaa34a2782211fadaeacb3395e33538d7c701c8ac8e492c: Status 404 returned error can't find the container with id 2cc8d68839ea61acdbaa34a2782211fadaeacb3395e33538d7c701c8ac8e492c Jul 20 02:55:06.950280 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.950194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:55:06.950486 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.950468 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 02:55:06.958780 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.958753 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:55:06.959145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:06.959126 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 02:55:07.422879 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:07.422836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" event={"ID":"1b6c66e2-5522-4894-a420-3e0381a84cbf","Type":"ContainerStarted","Data":"2cc8d68839ea61acdbaa34a2782211fadaeacb3395e33538d7c701c8ac8e492c"} Jul 20 02:55:09.433375 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.433346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" event={"ID":"abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d","Type":"ContainerStarted","Data":"7f9f4984d9b758766bfec5191b31e447131ca09e7ec7b22b0e5887324b3f4350"} Jul 20 02:55:09.433839 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.433493 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:09.434722 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.434701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" event={"ID":"1b6c66e2-5522-4894-a420-3e0381a84cbf","Type":"ContainerStarted","Data":"e60335544372558ca59df2dfbb0eb6c25270ee86483ccbf36cae9d646cb91831"} Jul 20 02:55:09.434829 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.434819 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:09.449667 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.449620 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" podStartSLOduration=1.954485708 podStartE2EDuration="5.449601576s" totalImagesPullingTime="3.495115868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:09.448590782 +0000 UTC m=+604.324392502" watchObservedRunningTime="2026-07-20 02:55:09.449601576 +0000 UTC m=+604.325403315" Jul 20 02:55:09.463374 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.463332 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" podStartSLOduration=3.213839741 podStartE2EDuration="5.463319998s" totalImagesPullingTime="2.249480257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:09.461842401 +0000 UTC m=+604.337644111" watchObservedRunningTime="2026-07-20 02:55:09.463319998 +0000 UTC m=+604.339121702" Jul 20 02:55:09.793189 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.793111 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:55:09.793338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:09.793307 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" containerID="cri-o://13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd" gracePeriod=30 Jul 20 02:55:10.034788 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.034765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:55:10.113375 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrmth\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113566 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113389 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113566 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113413 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113566 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113528 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113748 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113748 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.113748 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.113672 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert\") pod \"4514f897-672c-4dc0-8172-fafa747b9395\" (UID: \"4514f897-672c-4dc0-8172-fafa747b9395\") " Jul 20 02:55:10.114053 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.114023 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:55:10.115755 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.115731 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:55:10.115915 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.115891 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:55:10.115979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.115955 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs" (OuterVolumeSpecName: "local-certs") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:55:10.116059 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.116042 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token" (OuterVolumeSpecName: "istio-token") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:55:10.116125 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.116108 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts" (OuterVolumeSpecName: "cacerts") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:55:10.116160 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.116132 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth" (OuterVolumeSpecName: "kube-api-access-xrmth") pod "4514f897-672c-4dc0-8172-fafa747b9395" (UID: "4514f897-672c-4dc0-8172-fafa747b9395"). InnerVolumeSpecName "kube-api-access-xrmth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:55:10.214504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214469 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-dns-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214497 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmth\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-kube-api-access-xrmth\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214508 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4514f897-672c-4dc0-8172-fafa747b9395-local-certs\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214728 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214517 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-istio-kubeconfig\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214728 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214525 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4514f897-672c-4dc0-8172-fafa747b9395-istio-token\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214728 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214533 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4514f897-672c-4dc0-8172-fafa747b9395-istio-csr-ca-configmap\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.214728 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.214542 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4514f897-672c-4dc0-8172-fafa747b9395-cacerts\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:10.440553 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.440475 2565 generic.go:356] "Generic (PLEG): container finished" podID="4514f897-672c-4dc0-8172-fafa747b9395" containerID="13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd" exitCode=0 Jul 20 02:55:10.440553 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.440520 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" Jul 20 02:55:10.441040 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.440560 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" event={"ID":"4514f897-672c-4dc0-8172-fafa747b9395","Type":"ContainerDied","Data":"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd"} Jul 20 02:55:10.441040 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.440609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt" event={"ID":"4514f897-672c-4dc0-8172-fafa747b9395","Type":"ContainerDied","Data":"7ba9436258003f742a44b150ba94d9e5c566a9a52bc7753c9eafb7d38dbcb272"} Jul 20 02:55:10.441040 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.440634 2565 scope.go:117] "RemoveContainer" containerID="13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd" Jul 20 02:55:10.449495 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.449475 2565 scope.go:117] "RemoveContainer" containerID="13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd" Jul 20 02:55:10.449731 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:10.449711 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd\": container with ID starting with 13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd not found: ID does not exist" containerID="13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd" Jul 20 02:55:10.449792 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.449738 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd"} err="failed to get container status \"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd\": rpc error: code = NotFound desc = could not find container \"13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd\": container with ID starting with 13348fc647bc7688c94ba24d60dc5fa8ac6fc8bbb5d99cff212846d35af4b9cd not found: ID does not exist" Jul 20 02:55:10.461599 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.461577 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:55:10.468003 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:10.467984 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-tn6wt"] Jul 20 02:55:11.190959 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:11.190858 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4514f897-672c-4dc0-8172-fafa747b9395" path="/var/lib/kubelet/pods/4514f897-672c-4dc0-8172-fafa747b9395/volumes" Jul 20 02:55:40.443426 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:40.443393 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5msd8" Jul 20 02:55:40.446636 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:40.446609 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb9bv" Jul 20 02:55:55.564822 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.564764 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:55:55.565194 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.565082 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" Jul 20 02:55:55.565194 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.565092 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" Jul 20 02:55:55.565194 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.565153 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4514f897-672c-4dc0-8172-fafa747b9395" containerName="discovery" Jul 20 02:55:55.571890 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.571872 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:55.574963 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.574834 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 02:55:55.574963 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.574834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-m74q7\"" Jul 20 02:55:55.574963 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.574834 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 02:55:55.576627 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.576603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:55:55.688502 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.688463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtjr\" (UniqueName: \"kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr\") pod \"kuadrant-operator-catalog-99fdt\" (UID: \"71ba8a4f-9504-4a48-934e-502bf51cb59c\") " pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:55.789376 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.789337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtjr\" (UniqueName: \"kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr\") pod \"kuadrant-operator-catalog-99fdt\" (UID: \"71ba8a4f-9504-4a48-934e-502bf51cb59c\") " pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:55.796853 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.796824 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtjr\" (UniqueName: \"kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr\") pod \"kuadrant-operator-catalog-99fdt\" (UID: \"71ba8a4f-9504-4a48-934e-502bf51cb59c\") " pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:55.882752 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.882716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:55.938266 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:55.938237 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:55:56.007192 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.007140 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:55:56.009409 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:55:56.009383 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ba8a4f_9504_4a48_934e_502bf51cb59c.slice/crio-95c54ea31a4ce176dce722d71ed56fd1a9a436dafa5be69dd799fe462ae38cb8 WatchSource:0}: Error finding container 95c54ea31a4ce176dce722d71ed56fd1a9a436dafa5be69dd799fe462ae38cb8: Status 404 returned error can't find the container with id 95c54ea31a4ce176dce722d71ed56fd1a9a436dafa5be69dd799fe462ae38cb8 Jul 20 02:55:56.149351 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.149261 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fwqxc"] Jul 20 02:55:56.151961 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.151944 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:55:56.159733 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.159711 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fwqxc"] Jul 20 02:55:56.191994 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.191966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9br5v\" (UniqueName: \"kubernetes.io/projected/cd8d2a4b-5afa-4939-b397-65b73b57840a-kube-api-access-9br5v\") pod \"kuadrant-operator-catalog-fwqxc\" (UID: \"cd8d2a4b-5afa-4939-b397-65b73b57840a\") " pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:55:56.292836 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.292800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br5v\" (UniqueName: \"kubernetes.io/projected/cd8d2a4b-5afa-4939-b397-65b73b57840a-kube-api-access-9br5v\") pod \"kuadrant-operator-catalog-fwqxc\" (UID: \"cd8d2a4b-5afa-4939-b397-65b73b57840a\") " pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:55:56.301358 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.301327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br5v\" (UniqueName: \"kubernetes.io/projected/cd8d2a4b-5afa-4939-b397-65b73b57840a-kube-api-access-9br5v\") pod \"kuadrant-operator-catalog-fwqxc\" (UID: \"cd8d2a4b-5afa-4939-b397-65b73b57840a\") " pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:55:56.461445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.461354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:55:56.615268 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.615245 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fwqxc"] Jul 20 02:55:56.636147 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:55:56.636116 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8d2a4b_5afa_4939_b397_65b73b57840a.slice/crio-248bd5c311c080895ac8a8fe853cb935b0935404f71579dde829be596988480b WatchSource:0}: Error finding container 248bd5c311c080895ac8a8fe853cb935b0935404f71579dde829be596988480b: Status 404 returned error can't find the container with id 248bd5c311c080895ac8a8fe853cb935b0935404f71579dde829be596988480b Jul 20 02:55:56.640781 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:56.640741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" event={"ID":"71ba8a4f-9504-4a48-934e-502bf51cb59c","Type":"ContainerStarted","Data":"95c54ea31a4ce176dce722d71ed56fd1a9a436dafa5be69dd799fe462ae38cb8"} Jul 20 02:55:57.647537 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:57.647501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" event={"ID":"cd8d2a4b-5afa-4939-b397-65b73b57840a","Type":"ContainerStarted","Data":"248bd5c311c080895ac8a8fe853cb935b0935404f71579dde829be596988480b"} Jul 20 02:55:58.653748 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.653708 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" event={"ID":"cd8d2a4b-5afa-4939-b397-65b73b57840a","Type":"ContainerStarted","Data":"485e29d24d300282dc5917b68550b5e513dfce4d0239c6dbca89e2c571b9137c"} Jul 20 02:55:58.655211 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.655186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" event={"ID":"71ba8a4f-9504-4a48-934e-502bf51cb59c","Type":"ContainerStarted","Data":"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87"} Jul 20 02:55:58.655342 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.655268 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" podUID="71ba8a4f-9504-4a48-934e-502bf51cb59c" containerName="registry-server" containerID="cri-o://965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87" gracePeriod=2 Jul 20 02:55:58.666945 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.666892 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" podStartSLOduration=1.349588644 podStartE2EDuration="2.666873118s" totalImagesPullingTime="1.317284474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:58.66680359 +0000 UTC m=+653.542605302" watchObservedRunningTime="2026-07-20 02:55:58.666873118 +0000 UTC m=+653.542674829" Jul 20 02:55:58.680164 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.680121 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" podStartSLOduration=1.7368944210000001 podStartE2EDuration="3.680106321s" totalImagesPullingTime="1.9432119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:58.679594287 +0000 UTC m=+653.555396003" watchObservedRunningTime="2026-07-20 02:55:58.680106321 +0000 UTC m=+653.555908031" Jul 20 02:55:58.896132 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:58.896106 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:59.019272 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.019168 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxtjr\" (UniqueName: \"kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr\") pod \"71ba8a4f-9504-4a48-934e-502bf51cb59c\" (UID: \"71ba8a4f-9504-4a48-934e-502bf51cb59c\") " Jul 20 02:55:59.021286 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.021263 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr" (OuterVolumeSpecName: "kube-api-access-fxtjr") pod "71ba8a4f-9504-4a48-934e-502bf51cb59c" (UID: "71ba8a4f-9504-4a48-934e-502bf51cb59c"). InnerVolumeSpecName "kube-api-access-fxtjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:55:59.120068 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.120037 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxtjr\" (UniqueName: \"kubernetes.io/projected/71ba8a4f-9504-4a48-934e-502bf51cb59c-kube-api-access-fxtjr\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:55:59.661151 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.661116 2565 generic.go:356] "Generic (PLEG): container finished" podID="71ba8a4f-9504-4a48-934e-502bf51cb59c" containerID="965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87" exitCode=0 Jul 20 02:55:59.661543 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.661174 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" Jul 20 02:55:59.661543 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.661208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" event={"ID":"71ba8a4f-9504-4a48-934e-502bf51cb59c","Type":"ContainerDied","Data":"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87"} Jul 20 02:55:59.661543 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.661309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-99fdt" event={"ID":"71ba8a4f-9504-4a48-934e-502bf51cb59c","Type":"ContainerDied","Data":"95c54ea31a4ce176dce722d71ed56fd1a9a436dafa5be69dd799fe462ae38cb8"} Jul 20 02:55:59.661543 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.661336 2565 scope.go:117] "RemoveContainer" containerID="965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87" Jul 20 02:55:59.669687 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.669670 2565 scope.go:117] "RemoveContainer" containerID="965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87" Jul 20 02:55:59.669948 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:55:59.669928 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87\": container with ID starting with 965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87 not found: ID does not exist" containerID="965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87" Jul 20 02:55:59.669997 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.669955 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87"} err="failed to get container status \"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87\": rpc error: code = NotFound desc = could not find container \"965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87\": container with ID starting with 965cba3d20f98622ae7caf98368435f0ab609515cd32ab77f645baff39bbbb87 not found: ID does not exist" Jul 20 02:55:59.674878 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.674857 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:55:59.678396 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:55:59.678375 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-99fdt"] Jul 20 02:56:01.190639 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:01.190605 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ba8a4f-9504-4a48-934e-502bf51cb59c" path="/var/lib/kubelet/pods/71ba8a4f-9504-4a48-934e-502bf51cb59c/volumes" Jul 20 02:56:06.462395 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:06.462357 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:56:06.462874 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:06.462428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:56:06.484934 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:06.484908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:56:06.713616 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:06.713541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fwqxc" Jul 20 02:56:11.382007 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.381976 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt"] Jul 20 02:56:11.382445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.382261 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ba8a4f-9504-4a48-934e-502bf51cb59c" containerName="registry-server" Jul 20 02:56:11.382445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.382271 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ba8a4f-9504-4a48-934e-502bf51cb59c" containerName="registry-server" Jul 20 02:56:11.382445 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.382349 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ba8a4f-9504-4a48-934e-502bf51cb59c" containerName="registry-server" Jul 20 02:56:11.386505 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.386489 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.388452 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.388428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2nxdn\"" Jul 20 02:56:11.392032 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.392009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt"] Jul 20 02:56:11.524216 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.524176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2hj\" (UniqueName: \"kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.524507 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.524242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.524507 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.524342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.625148 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.625110 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.625148 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.625153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.625444 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.625195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2hj\" (UniqueName: \"kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.625592 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.625568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.625660 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.625594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.632592 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.632533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2hj\" (UniqueName: \"kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.696520 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.696471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:11.826065 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.826040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt"] Jul 20 02:56:11.827712 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:11.827680 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4270937e_3f48_42a7_a4d7_253dccb04da0.slice/crio-47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f WatchSource:0}: Error finding container 47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f: Status 404 returned error can't find the container with id 47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f Jul 20 02:56:11.981012 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.980975 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh"] Jul 20 02:56:11.984328 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.984283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:11.991103 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:11.991080 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh"] Jul 20 02:56:12.129637 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.129605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.129809 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.129648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5rm\" (UniqueName: \"kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.129809 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.129673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.230777 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.230700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5rm\" (UniqueName: \"kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.230777 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.230737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.230937 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.230799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.231121 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.231104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.231160 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.231131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.237772 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.237747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5rm\" (UniqueName: \"kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.306436 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.306402 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:12.431030 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.430946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh"] Jul 20 02:56:12.433333 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:12.433304 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fc172c_2df2_4131_8be7_17d294d0b13e.slice/crio-8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419 WatchSource:0}: Error finding container 8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419: Status 404 returned error can't find the container with id 8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419 Jul 20 02:56:12.610284 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.610245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq"] Jul 20 02:56:12.613720 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.613703 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.622025 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.621997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq"] Jul 20 02:56:12.719960 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.719931 2565 generic.go:356] "Generic (PLEG): container finished" podID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerID="954be6f019a848bd527f3dd4de953e0a8bb584484cba8489a82630aba7b00fda" exitCode=0 Jul 20 02:56:12.720154 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.720022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" event={"ID":"07fc172c-2df2-4131-8be7-17d294d0b13e","Type":"ContainerDied","Data":"954be6f019a848bd527f3dd4de953e0a8bb584484cba8489a82630aba7b00fda"} Jul 20 02:56:12.720154 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.720074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" event={"ID":"07fc172c-2df2-4131-8be7-17d294d0b13e","Type":"ContainerStarted","Data":"8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419"} Jul 20 02:56:12.721590 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.721568 2565 generic.go:356] "Generic (PLEG): container finished" podID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerID="1e20b5af9116cb7f2cc7a7ae63c558249714a6b38fe222c12ae143e272627d2a" exitCode=0 Jul 20 02:56:12.721677 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.721603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" event={"ID":"4270937e-3f48-42a7-a4d7-253dccb04da0","Type":"ContainerDied","Data":"1e20b5af9116cb7f2cc7a7ae63c558249714a6b38fe222c12ae143e272627d2a"} Jul 20 02:56:12.721677 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.721642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" event={"ID":"4270937e-3f48-42a7-a4d7-253dccb04da0","Type":"ContainerStarted","Data":"47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f"} Jul 20 02:56:12.734831 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.734758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.734831 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.734813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlk5\" (UniqueName: \"kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.734987 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.734901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.835350 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.835278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.835350 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.835358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.835558 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.835391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlk5\" (UniqueName: \"kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.835769 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.835745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.835769 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.835761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.842650 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.842629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlk5\" (UniqueName: \"kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:12.923763 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:12.923735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:13.068757 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.068732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq"] Jul 20 02:56:13.070749 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:13.070724 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c4704e_51cd_482b_9207_781a3d502a9c.slice/crio-237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1 WatchSource:0}: Error finding container 237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1: Status 404 returned error can't find the container with id 237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1 Jul 20 02:56:13.181864 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.181832 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc"] Jul 20 02:56:13.190272 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.190164 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.199699 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.199670 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc"] Jul 20 02:56:13.338415 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.338383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.338634 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.338429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.338634 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.338515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsm97\" (UniqueName: \"kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.439966 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.439925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.439966 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.439971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.440506 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.440009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsm97\" (UniqueName: \"kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.440506 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.440372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.440506 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.440369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.447335 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.447282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsm97\" (UniqueName: \"kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.529024 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.529003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:13.666348 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.666322 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc"] Jul 20 02:56:13.677264 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:13.677229 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c2b239_dc2f_4caa_9029_f47e09b1f32d.slice/crio-4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5 WatchSource:0}: Error finding container 4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5: Status 404 returned error can't find the container with id 4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5 Jul 20 02:56:13.728260 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.728228 2565 generic.go:356] "Generic (PLEG): container finished" podID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerID="2f8712e100795a3aa8dc6645364a9d57cf89bf7e05ee5d2dc168c95470755265" exitCode=0 Jul 20 02:56:13.728435 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.728304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" event={"ID":"4270937e-3f48-42a7-a4d7-253dccb04da0","Type":"ContainerDied","Data":"2f8712e100795a3aa8dc6645364a9d57cf89bf7e05ee5d2dc168c95470755265"} Jul 20 02:56:13.733635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.730039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" event={"ID":"d5c2b239-dc2f-4caa-9029-f47e09b1f32d","Type":"ContainerStarted","Data":"4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5"} Jul 20 02:56:13.736893 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.736864 2565 generic.go:356] "Generic (PLEG): container finished" podID="80c4704e-51cd-482b-9207-781a3d502a9c" containerID="cd67a1275333d3b732a8324529d0c52c210f553aafb56b4f3282e0fd29b761e4" exitCode=0 Jul 20 02:56:13.736983 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.736943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" event={"ID":"80c4704e-51cd-482b-9207-781a3d502a9c","Type":"ContainerDied","Data":"cd67a1275333d3b732a8324529d0c52c210f553aafb56b4f3282e0fd29b761e4"} Jul 20 02:56:13.737039 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.736988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" event={"ID":"80c4704e-51cd-482b-9207-781a3d502a9c","Type":"ContainerStarted","Data":"237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1"} Jul 20 02:56:13.739094 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.739068 2565 generic.go:356] "Generic (PLEG): container finished" podID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerID="c6fe0b5eb14883837108e657324b4b363d674d0e1f03e765a8c73b6905a06f57" exitCode=0 Jul 20 02:56:13.739198 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:13.739106 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" event={"ID":"07fc172c-2df2-4131-8be7-17d294d0b13e","Type":"ContainerDied","Data":"c6fe0b5eb14883837108e657324b4b363d674d0e1f03e765a8c73b6905a06f57"} Jul 20 02:56:14.745626 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.745587 2565 generic.go:356] "Generic (PLEG): container finished" podID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerID="d2b0b370e074f484fe983a2c729ddc82159349be723795bcc55d9d45ade7e18a" exitCode=0 Jul 20 02:56:14.746086 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.745659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" event={"ID":"d5c2b239-dc2f-4caa-9029-f47e09b1f32d","Type":"ContainerDied","Data":"d2b0b370e074f484fe983a2c729ddc82159349be723795bcc55d9d45ade7e18a"} Jul 20 02:56:14.747576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.747546 2565 generic.go:356] "Generic (PLEG): container finished" podID="80c4704e-51cd-482b-9207-781a3d502a9c" containerID="d0c0c31d3f33779666c5022056f9c3aa893ed6903afb78f4a6592c8e9a9071e5" exitCode=0 Jul 20 02:56:14.747702 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.747674 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" event={"ID":"80c4704e-51cd-482b-9207-781a3d502a9c","Type":"ContainerDied","Data":"d0c0c31d3f33779666c5022056f9c3aa893ed6903afb78f4a6592c8e9a9071e5"} Jul 20 02:56:14.749998 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.749977 2565 generic.go:356] "Generic (PLEG): container finished" podID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerID="f0db8438fb595622e83bbe1bc116cccc685c60b5d1b0aa81f614ae015dfeec96" exitCode=0 Jul 20 02:56:14.750105 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.750063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" event={"ID":"07fc172c-2df2-4131-8be7-17d294d0b13e","Type":"ContainerDied","Data":"f0db8438fb595622e83bbe1bc116cccc685c60b5d1b0aa81f614ae015dfeec96"} Jul 20 02:56:14.752215 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.752193 2565 generic.go:356] "Generic (PLEG): container finished" podID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerID="7e01e2397e75c517deef1a2541da3a8e1c45860b4dcfa0c33affb5a97f7dc32e" exitCode=0 Jul 20 02:56:14.752346 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:14.752228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" event={"ID":"4270937e-3f48-42a7-a4d7-253dccb04da0","Type":"ContainerDied","Data":"7e01e2397e75c517deef1a2541da3a8e1c45860b4dcfa0c33affb5a97f7dc32e"} Jul 20 02:56:15.758133 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.758040 2565 generic.go:356] "Generic (PLEG): container finished" podID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerID="c225e4ba894a2fba75c94eca4b837e2ec0ccadf0e5578035ced43392c1a33c65" exitCode=0 Jul 20 02:56:15.758557 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.758121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" event={"ID":"d5c2b239-dc2f-4caa-9029-f47e09b1f32d","Type":"ContainerDied","Data":"c225e4ba894a2fba75c94eca4b837e2ec0ccadf0e5578035ced43392c1a33c65"} Jul 20 02:56:15.760368 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.760346 2565 generic.go:356] "Generic (PLEG): container finished" podID="80c4704e-51cd-482b-9207-781a3d502a9c" containerID="2c7f5cb1a581fc50a1e11ea7176906e415582f35c448caa89c100c82fdb9e471" exitCode=0 Jul 20 02:56:15.760479 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.760456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" event={"ID":"80c4704e-51cd-482b-9207-781a3d502a9c","Type":"ContainerDied","Data":"2c7f5cb1a581fc50a1e11ea7176906e415582f35c448caa89c100c82fdb9e471"} Jul 20 02:56:15.905835 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.905809 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:15.978182 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:15.978155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:16.062617 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062535 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr2hj\" (UniqueName: \"kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj\") pod \"4270937e-3f48-42a7-a4d7-253dccb04da0\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " Jul 20 02:56:16.062617 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062565 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle\") pod \"07fc172c-2df2-4131-8be7-17d294d0b13e\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " Jul 20 02:56:16.062617 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062615 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util\") pod \"4270937e-3f48-42a7-a4d7-253dccb04da0\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " Jul 20 02:56:16.062866 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h5rm\" (UniqueName: \"kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm\") pod \"07fc172c-2df2-4131-8be7-17d294d0b13e\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " Jul 20 02:56:16.062866 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062678 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle\") pod \"4270937e-3f48-42a7-a4d7-253dccb04da0\" (UID: \"4270937e-3f48-42a7-a4d7-253dccb04da0\") " Jul 20 02:56:16.062866 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.062697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util\") pod \"07fc172c-2df2-4131-8be7-17d294d0b13e\" (UID: \"07fc172c-2df2-4131-8be7-17d294d0b13e\") " Jul 20 02:56:16.063211 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.063169 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle" (OuterVolumeSpecName: "bundle") pod "07fc172c-2df2-4131-8be7-17d294d0b13e" (UID: "07fc172c-2df2-4131-8be7-17d294d0b13e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:16.063479 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.063451 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle" (OuterVolumeSpecName: "bundle") pod "4270937e-3f48-42a7-a4d7-253dccb04da0" (UID: "4270937e-3f48-42a7-a4d7-253dccb04da0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:16.064781 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.064753 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj" (OuterVolumeSpecName: "kube-api-access-hr2hj") pod "4270937e-3f48-42a7-a4d7-253dccb04da0" (UID: "4270937e-3f48-42a7-a4d7-253dccb04da0"). InnerVolumeSpecName "kube-api-access-hr2hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:56:16.064885 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.064775 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm" (OuterVolumeSpecName: "kube-api-access-6h5rm") pod "07fc172c-2df2-4131-8be7-17d294d0b13e" (UID: "07fc172c-2df2-4131-8be7-17d294d0b13e"). InnerVolumeSpecName "kube-api-access-6h5rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:56:16.071599 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.071556 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util" (OuterVolumeSpecName: "util") pod "07fc172c-2df2-4131-8be7-17d294d0b13e" (UID: "07fc172c-2df2-4131-8be7-17d294d0b13e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:16.071753 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.071734 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util" (OuterVolumeSpecName: "util") pod "4270937e-3f48-42a7-a4d7-253dccb04da0" (UID: "4270937e-3f48-42a7-a4d7-253dccb04da0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:16.163930 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163895 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.163930 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163922 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5rm\" (UniqueName: \"kubernetes.io/projected/07fc172c-2df2-4131-8be7-17d294d0b13e-kube-api-access-6h5rm\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.163930 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163932 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4270937e-3f48-42a7-a4d7-253dccb04da0-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.163930 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163940 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.164223 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163948 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2hj\" (UniqueName: \"kubernetes.io/projected/4270937e-3f48-42a7-a4d7-253dccb04da0-kube-api-access-hr2hj\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.164223 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.163956 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fc172c-2df2-4131-8be7-17d294d0b13e-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:16.766953 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.766924 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" Jul 20 02:56:16.767385 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.766923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh" event={"ID":"07fc172c-2df2-4131-8be7-17d294d0b13e","Type":"ContainerDied","Data":"8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419"} Jul 20 02:56:16.767385 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.767024 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8237cd309a63f6c0bf9fbf269859c47b9313a3bbfa397fd8eed0852d4696b419" Jul 20 02:56:16.768896 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.768867 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" Jul 20 02:56:16.769020 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.768888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt" event={"ID":"4270937e-3f48-42a7-a4d7-253dccb04da0","Type":"ContainerDied","Data":"47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f"} Jul 20 02:56:16.769020 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.768935 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b49438ee01413310994abe27251d82259a22cfe1b99faf5697f745addfa18f" Jul 20 02:56:16.771062 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.771035 2565 generic.go:356] "Generic (PLEG): container finished" podID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerID="567af006142b6d113f3ac389db516df2bd91c117c461f77fb42ed697f8b9bc6c" exitCode=0 Jul 20 02:56:16.771178 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.771125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" event={"ID":"d5c2b239-dc2f-4caa-9029-f47e09b1f32d","Type":"ContainerDied","Data":"567af006142b6d113f3ac389db516df2bd91c117c461f77fb42ed697f8b9bc6c"} Jul 20 02:56:16.899247 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.899225 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:16.970366 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.970335 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle\") pod \"80c4704e-51cd-482b-9207-781a3d502a9c\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " Jul 20 02:56:16.970524 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.970431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util\") pod \"80c4704e-51cd-482b-9207-781a3d502a9c\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " Jul 20 02:56:16.970524 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.970460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptlk5\" (UniqueName: \"kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5\") pod \"80c4704e-51cd-482b-9207-781a3d502a9c\" (UID: \"80c4704e-51cd-482b-9207-781a3d502a9c\") " Jul 20 02:56:16.970968 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.970942 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle" (OuterVolumeSpecName: "bundle") pod "80c4704e-51cd-482b-9207-781a3d502a9c" (UID: "80c4704e-51cd-482b-9207-781a3d502a9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:16.972515 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.972494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5" (OuterVolumeSpecName: "kube-api-access-ptlk5") pod "80c4704e-51cd-482b-9207-781a3d502a9c" (UID: "80c4704e-51cd-482b-9207-781a3d502a9c"). InnerVolumeSpecName "kube-api-access-ptlk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:56:16.978086 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:16.978062 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util" (OuterVolumeSpecName: "util") pod "80c4704e-51cd-482b-9207-781a3d502a9c" (UID: "80c4704e-51cd-482b-9207-781a3d502a9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:17.071320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.071215 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:17.071320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.071244 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80c4704e-51cd-482b-9207-781a3d502a9c-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:17.071320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.071253 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptlk5\" (UniqueName: \"kubernetes.io/projected/80c4704e-51cd-482b-9207-781a3d502a9c-kube-api-access-ptlk5\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:17.777632 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.777607 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" Jul 20 02:56:17.777632 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.777618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq" event={"ID":"80c4704e-51cd-482b-9207-781a3d502a9c","Type":"ContainerDied","Data":"237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1"} Jul 20 02:56:17.778062 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.777660 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="237a2d4731f21d0ec283d096d892b7bdac286707f8deacdd045b00dec93860a1" Jul 20 02:56:17.905046 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.905022 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:17.979005 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.978976 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util\") pod \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " Jul 20 02:56:17.979162 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.979025 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle\") pod \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " Jul 20 02:56:17.979162 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.979058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsm97\" (UniqueName: \"kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97\") pod \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\" (UID: \"d5c2b239-dc2f-4caa-9029-f47e09b1f32d\") " Jul 20 02:56:17.979615 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.979588 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle" (OuterVolumeSpecName: "bundle") pod "d5c2b239-dc2f-4caa-9029-f47e09b1f32d" (UID: "d5c2b239-dc2f-4caa-9029-f47e09b1f32d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:17.981195 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.981164 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97" (OuterVolumeSpecName: "kube-api-access-jsm97") pod "d5c2b239-dc2f-4caa-9029-f47e09b1f32d" (UID: "d5c2b239-dc2f-4caa-9029-f47e09b1f32d"). InnerVolumeSpecName "kube-api-access-jsm97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:56:17.984708 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:17.984675 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util" (OuterVolumeSpecName: "util") pod "d5c2b239-dc2f-4caa-9029-f47e09b1f32d" (UID: "d5c2b239-dc2f-4caa-9029-f47e09b1f32d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:18.079589 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.079559 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-util\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:18.079589 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.079590 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-bundle\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:18.079761 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.079600 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsm97\" (UniqueName: \"kubernetes.io/projected/d5c2b239-dc2f-4caa-9029-f47e09b1f32d-kube-api-access-jsm97\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:18.784040 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.784011 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" Jul 20 02:56:18.784040 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.784017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc" event={"ID":"d5c2b239-dc2f-4caa-9029-f47e09b1f32d","Type":"ContainerDied","Data":"4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5"} Jul 20 02:56:18.784472 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:18.784098 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4978a22b8d3bab2b4e945d2d2c2be3d08f5409ec134e00fd1bd1786952b1f2f5" Jul 20 02:56:19.974821 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.974788 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78f8b58b84-vmqph"] Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975067 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975077 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975087 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975092 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975104 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975110 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975116 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975120 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975129 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975134 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975139 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975144 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975150 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975155 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975164 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975169 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975174 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975179 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975184 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975189 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="pull" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975196 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975210 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="util" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975217 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="extract" Jul 20 02:56:19.975213 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975223 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="extract" Jul 20 02:56:19.976706 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975267 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c4704e-51cd-482b-9207-781a3d502a9c" containerName="extract" Jul 20 02:56:19.976706 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975279 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fc172c-2df2-4131-8be7-17d294d0b13e" containerName="extract" Jul 20 02:56:19.976706 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975286 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4270937e-3f48-42a7-a4d7-253dccb04da0" containerName="extract" Jul 20 02:56:19.976706 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.975309 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c2b239-dc2f-4caa-9029-f47e09b1f32d" containerName="extract" Jul 20 02:56:19.978803 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.978782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:19.980762 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.980742 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 02:56:19.980881 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.980864 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 02:56:19.980920 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.980865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5q4k\"" Jul 20 02:56:19.981701 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.981551 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 02:56:19.981701 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.981572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 02:56:19.981701 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.981597 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 02:56:19.981701 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.981563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 02:56:19.981701 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.981664 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 02:56:19.986066 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.986045 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 02:56:19.986538 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:19.986328 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f8b58b84-vmqph"] Jul 20 02:56:20.093007 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.092971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-oauth-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093007 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-service-ca\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-oauth-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-trusted-ca-bundle\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-console-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.093212 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.093162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nftp\" (UniqueName: \"kubernetes.io/projected/3440a165-5182-437a-ab26-85e6f7d16654-kube-api-access-5nftp\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194271 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-oauth-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194271 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-service-ca\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-oauth-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-trusted-ca-bundle\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-console-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.194680 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.194483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nftp\" (UniqueName: \"kubernetes.io/projected/3440a165-5182-437a-ab26-85e6f7d16654-kube-api-access-5nftp\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.195066 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.195039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-oauth-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.195167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.195102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-service-ca\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.195221 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.195209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-trusted-ca-bundle\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.195263 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.195241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3440a165-5182-437a-ab26-85e6f7d16654-console-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.196859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.196838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-serving-cert\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.196980 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.196966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3440a165-5182-437a-ab26-85e6f7d16654-console-oauth-config\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.202074 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.202055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nftp\" (UniqueName: \"kubernetes.io/projected/3440a165-5182-437a-ab26-85e6f7d16654-kube-api-access-5nftp\") pod \"console-78f8b58b84-vmqph\" (UID: \"3440a165-5182-437a-ab26-85e6f7d16654\") " pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.290477 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.290396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:20.417004 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.416979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f8b58b84-vmqph"] Jul 20 02:56:20.418553 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:20.418516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3440a165_5182_437a_ab26_85e6f7d16654.slice/crio-4e0118e50cc321c99d1c156a5de3206545f385049528a61e44f13a57627c2f60 WatchSource:0}: Error finding container 4e0118e50cc321c99d1c156a5de3206545f385049528a61e44f13a57627c2f60: Status 404 returned error can't find the container with id 4e0118e50cc321c99d1c156a5de3206545f385049528a61e44f13a57627c2f60 Jul 20 02:56:20.795612 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.795576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f8b58b84-vmqph" event={"ID":"3440a165-5182-437a-ab26-85e6f7d16654","Type":"ContainerStarted","Data":"295beb2ea4fd0863c9e75d225ad34cfcb97f4a64c4d19e776a112da736111f0b"} Jul 20 02:56:20.795764 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.795626 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f8b58b84-vmqph" event={"ID":"3440a165-5182-437a-ab26-85e6f7d16654","Type":"ContainerStarted","Data":"4e0118e50cc321c99d1c156a5de3206545f385049528a61e44f13a57627c2f60"} Jul 20 02:56:20.811822 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:20.811776 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78f8b58b84-vmqph" podStartSLOduration=1.811761319 podStartE2EDuration="1.811761319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:20.811133726 +0000 UTC m=+675.686935436" watchObservedRunningTime="2026-07-20 02:56:20.811761319 +0000 UTC m=+675.687563029" Jul 20 02:56:27.801772 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.801734 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn"] Jul 20 02:56:27.808653 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.808636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:27.810852 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.810821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 20 02:56:27.811147 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.811129 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s6q6d\"" Jul 20 02:56:27.814104 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.814071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn"] Jul 20 02:56:27.961068 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:27.961028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwnb\" (UniqueName: \"kubernetes.io/projected/77435ef7-6ddc-4b38-9dde-78c52cf8767e-kube-api-access-2nwnb\") pod \"dns-operator-controller-manager-864dd78c4c-8btkn\" (UID: \"77435ef7-6ddc-4b38-9dde-78c52cf8767e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:28.062057 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.061960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwnb\" (UniqueName: \"kubernetes.io/projected/77435ef7-6ddc-4b38-9dde-78c52cf8767e-kube-api-access-2nwnb\") pod \"dns-operator-controller-manager-864dd78c4c-8btkn\" (UID: \"77435ef7-6ddc-4b38-9dde-78c52cf8767e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:28.070888 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.070855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwnb\" (UniqueName: \"kubernetes.io/projected/77435ef7-6ddc-4b38-9dde-78c52cf8767e-kube-api-access-2nwnb\") pod \"dns-operator-controller-manager-864dd78c4c-8btkn\" (UID: \"77435ef7-6ddc-4b38-9dde-78c52cf8767e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:28.120011 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.119976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:28.266550 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.266523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn"] Jul 20 02:56:28.268231 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:28.268202 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77435ef7_6ddc_4b38_9dde_78c52cf8767e.slice/crio-5b3377c55cece97fa6571f468a22a6d360a86d26df6ee5965bd72b7fac426178 WatchSource:0}: Error finding container 5b3377c55cece97fa6571f468a22a6d360a86d26df6ee5965bd72b7fac426178: Status 404 returned error can't find the container with id 5b3377c55cece97fa6571f468a22a6d360a86d26df6ee5965bd72b7fac426178 Jul 20 02:56:28.270132 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.270111 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:56:28.842343 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:28.842310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" event={"ID":"77435ef7-6ddc-4b38-9dde-78c52cf8767e","Type":"ContainerStarted","Data":"5b3377c55cece97fa6571f468a22a6d360a86d26df6ee5965bd72b7fac426178"} Jul 20 02:56:30.290539 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.290499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:30.290906 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.290558 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:30.295276 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.295254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:30.855123 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.855083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" event={"ID":"77435ef7-6ddc-4b38-9dde-78c52cf8767e","Type":"ContainerStarted","Data":"7aa09b7537e20744c2a08e63136ca715c29faa36f920e8b12b28f373a1951545"} Jul 20 02:56:30.855339 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.855322 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:30.859153 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.859132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78f8b58b84-vmqph" Jul 20 02:56:30.870647 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:30.870600 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" podStartSLOduration=1.978132655 podStartE2EDuration="3.870587029s" totalImagesPullingTime="1.892454374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:30.868980401 +0000 UTC m=+685.744782110" watchObservedRunningTime="2026-07-20 02:56:30.870587029 +0000 UTC m=+685.746388738" Jul 20 02:56:32.711116 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.711077 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c"] Jul 20 02:56:32.717256 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.717235 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.719214 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.719196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c9sxh\"" Jul 20 02:56:32.724315 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.724212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c"] Jul 20 02:56:32.804593 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.804556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.804766 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.804607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55r9\" (UniqueName: \"kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.905327 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.905268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.905470 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.905360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55r9\" (UniqueName: \"kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.905647 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.905628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:32.922655 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:32.922628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55r9\" (UniqueName: \"kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:33.028137 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:33.028054 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:33.162916 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:33.162882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c"] Jul 20 02:56:33.166327 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:33.166283 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1b8f5b_e045_4b55_9991_70d6cf0ce719.slice/crio-2ebddb35ea9557ed14df30b314e2c25518b67793a7874bd6c51226b245e24f3b WatchSource:0}: Error finding container 2ebddb35ea9557ed14df30b314e2c25518b67793a7874bd6c51226b245e24f3b: Status 404 returned error can't find the container with id 2ebddb35ea9557ed14df30b314e2c25518b67793a7874bd6c51226b245e24f3b Jul 20 02:56:33.872935 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:33.872902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" event={"ID":"8f1b8f5b-e045-4b55-9991-70d6cf0ce719","Type":"ContainerStarted","Data":"2ebddb35ea9557ed14df30b314e2c25518b67793a7874bd6c51226b245e24f3b"} Jul 20 02:56:36.600573 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.600539 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh"] Jul 20 02:56:36.604325 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.604282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:36.606763 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.606743 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-92sln\"" Jul 20 02:56:36.615224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.615201 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh"] Jul 20 02:56:36.636437 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.636411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tgh\" (UniqueName: \"kubernetes.io/projected/1db3106f-5a3c-4ca2-a277-6ad3439e676b-kube-api-access-f7tgh\") pod \"authorino-operator-75cc8d5cb7-sz6gh\" (UID: \"1db3106f-5a3c-4ca2-a277-6ad3439e676b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:36.737041 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.736998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tgh\" (UniqueName: \"kubernetes.io/projected/1db3106f-5a3c-4ca2-a277-6ad3439e676b-kube-api-access-f7tgh\") pod \"authorino-operator-75cc8d5cb7-sz6gh\" (UID: \"1db3106f-5a3c-4ca2-a277-6ad3439e676b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:36.763600 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.763569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tgh\" (UniqueName: \"kubernetes.io/projected/1db3106f-5a3c-4ca2-a277-6ad3439e676b-kube-api-access-f7tgh\") pod \"authorino-operator-75cc8d5cb7-sz6gh\" (UID: \"1db3106f-5a3c-4ca2-a277-6ad3439e676b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:36.914462 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:36.914380 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:37.055906 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:37.055877 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh"] Jul 20 02:56:37.057914 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:37.057889 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db3106f_5a3c_4ca2_a277_6ad3439e676b.slice/crio-786c9ddd6c94fe752bc0dc576bb71de022377d714dda6b76892ab61434bac570 WatchSource:0}: Error finding container 786c9ddd6c94fe752bc0dc576bb71de022377d714dda6b76892ab61434bac570: Status 404 returned error can't find the container with id 786c9ddd6c94fe752bc0dc576bb71de022377d714dda6b76892ab61434bac570 Jul 20 02:56:37.894440 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:37.894394 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" event={"ID":"1db3106f-5a3c-4ca2-a277-6ad3439e676b","Type":"ContainerStarted","Data":"786c9ddd6c94fe752bc0dc576bb71de022377d714dda6b76892ab61434bac570"} Jul 20 02:56:38.903063 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:38.903022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" event={"ID":"8f1b8f5b-e045-4b55-9991-70d6cf0ce719","Type":"ContainerStarted","Data":"c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7"} Jul 20 02:56:38.903656 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:38.903116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:38.923820 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:38.923368 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" podStartSLOduration=1.636818162 podStartE2EDuration="6.923348286s" totalImagesPullingTime="5.286530124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:38.92078814 +0000 UTC m=+693.796589852" watchObservedRunningTime="2026-07-20 02:56:38.923348286 +0000 UTC m=+693.799149997" Jul 20 02:56:40.914726 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:40.914690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" event={"ID":"1db3106f-5a3c-4ca2-a277-6ad3439e676b","Type":"ContainerStarted","Data":"f16f05de937f442465fe181fd7bad384b5c3effa506de9dcc3b1324810d7d27d"} Jul 20 02:56:40.915077 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:40.914840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:40.935539 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:40.935491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" podStartSLOduration=1.80342533 podStartE2EDuration="4.935475367s" totalImagesPullingTime="3.132050037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:40.933195299 +0000 UTC m=+695.808997023" watchObservedRunningTime="2026-07-20 02:56:40.935475367 +0000 UTC m=+695.811277077" Jul 20 02:56:41.492414 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.492372 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c"] Jul 20 02:56:41.495816 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.495793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.497741 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.497712 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 20 02:56:41.497876 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.497779 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2nxdn\"" Jul 20 02:56:41.497876 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.497790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 20 02:56:41.501969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.501948 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c"] Jul 20 02:56:41.579776 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.579735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z68w\" (UniqueName: \"kubernetes.io/projected/26aa1c8f-e647-402d-8557-5cd153bfc98a-kube-api-access-5z68w\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.579954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.579830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.579954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.579875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26aa1c8f-e647-402d-8557-5cd153bfc98a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.680732 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.680699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z68w\" (UniqueName: \"kubernetes.io/projected/26aa1c8f-e647-402d-8557-5cd153bfc98a-kube-api-access-5z68w\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.680902 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.680750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.680902 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.680789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26aa1c8f-e647-402d-8557-5cd153bfc98a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.680902 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:41.680866 2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 20 02:56:41.681176 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:41.680929 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert podName:26aa1c8f-e647-402d-8557-5cd153bfc98a nodeName:}" failed. No retries permitted until 2026-07-20 02:56:42.180913398 +0000 UTC m=+697.056715089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jhn8c" (UID: "26aa1c8f-e647-402d-8557-5cd153bfc98a") : secret "plugin-serving-cert" not found Jul 20 02:56:41.681441 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.681423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26aa1c8f-e647-402d-8557-5cd153bfc98a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.688924 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.688900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z68w\" (UniqueName: \"kubernetes.io/projected/26aa1c8f-e647-402d-8557-5cd153bfc98a-kube-api-access-5z68w\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:41.865518 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:41.865487 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8btkn" Jul 20 02:56:42.184596 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:42.184502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:42.186993 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:42.186966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/26aa1c8f-e647-402d-8557-5cd153bfc98a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jhn8c\" (UID: \"26aa1c8f-e647-402d-8557-5cd153bfc98a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:42.406475 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:42.406443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" Jul 20 02:56:42.535871 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:42.535669 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c"] Jul 20 02:56:42.537946 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:56:42.537919 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26aa1c8f_e647_402d_8557_5cd153bfc98a.slice/crio-532486c80f94d6c7dfd9c4f302d1c745fe739755753ca75b0e8df7b3580bddc5 WatchSource:0}: Error finding container 532486c80f94d6c7dfd9c4f302d1c745fe739755753ca75b0e8df7b3580bddc5: Status 404 returned error can't find the container with id 532486c80f94d6c7dfd9c4f302d1c745fe739755753ca75b0e8df7b3580bddc5 Jul 20 02:56:42.932870 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:42.932835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" event={"ID":"26aa1c8f-e647-402d-8557-5cd153bfc98a","Type":"ContainerStarted","Data":"532486c80f94d6c7dfd9c4f302d1c745fe739755753ca75b0e8df7b3580bddc5"} Jul 20 02:56:49.910881 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:49.910845 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:51.474733 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.474699 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft"] Jul 20 02:56:51.502907 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.502872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft"] Jul 20 02:56:51.503111 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.503013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.572843 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.572776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwln\" (UniqueName: \"kubernetes.io/projected/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-kube-api-access-ddwln\") pod \"kuadrant-operator-controller-manager-86878c769f-c5mft\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.573060 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.572875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c5mft\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.575924 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.575896 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c"] Jul 20 02:56:51.576139 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.576114 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" containerName="manager" containerID="cri-o://c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7" gracePeriod=2 Jul 20 02:56:51.581886 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.581857 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c"] Jul 20 02:56:51.602358 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.602071 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft"] Jul 20 02:56:51.602716 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.602655 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-ddwln], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" Jul 20 02:56:51.605900 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.605870 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:56:51.606353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.606335 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" containerName="manager" Jul 20 02:56:51.606353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.606355 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" containerName="manager" Jul 20 02:56:51.606498 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.606447 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" containerName="manager" Jul 20 02:56:51.610736 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.610717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.613696 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.613513 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:51.614734 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.614710 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft"] Jul 20 02:56:51.627688 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.627658 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:56:51.642002 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.641970 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 02:56:51.647731 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.647709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.655794 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.655759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 02:56:51.667342 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.667278 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:51.673824 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.673795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.673984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.673870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftgs\" (UniqueName: \"kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.673984 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.673932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwln\" (UniqueName: \"kubernetes.io/projected/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-kube-api-access-ddwln\") pod \"kuadrant-operator-controller-manager-86878c769f-c5mft\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.674097 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.674034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c5mft\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.674495 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.674468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c5mft\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.677768 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.677743 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ddwln for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-c5mft" not found Jul 20 02:56:51.677875 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.677828 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-kube-api-access-ddwln podName:6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a nodeName:}" failed. No retries permitted until 2026-07-20 02:56:52.177808269 +0000 UTC m=+707.053609957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ddwln" (UniqueName: "kubernetes.io/projected/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-kube-api-access-ddwln") pod "kuadrant-operator-controller-manager-86878c769f-c5mft" (UID: "6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-c5mft" not found Jul 20 02:56:51.774912 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.774819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.774912 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.774880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftgs\" (UniqueName: \"kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.775147 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.774967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7s9\" (UniqueName: \"kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.775147 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.775034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.775455 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.775437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.782660 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.782631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftgs\" (UniqueName: \"kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nct46\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.876220 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.876172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7s9\" (UniqueName: \"kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.876424 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.876275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.876652 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.876630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.886859 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.886586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7s9\" (UniqueName: \"kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9\") pod \"kuadrant-operator-controller-manager-86878c769f-2hsgk\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.929811 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.929783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sz6gh" Jul 20 02:56:51.931639 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.931606 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:51.950428 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.950394 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:56:51.963353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.963322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:56:51.987793 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.987763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.990112 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.990073 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-c5mft\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.991777 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.991727 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:51.997805 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:51.997782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:51.999506 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:51.999477 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:52.000798 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:52.000773 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-c5mft\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:52.078000 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:52.077966 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume\") pod \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\" (UID: \"6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a\") " Jul 20 02:56:52.078166 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:52.078129 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwln\" (UniqueName: \"kubernetes.io/projected/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-kube-api-access-ddwln\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:52.078245 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:52.078219 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" (UID: "6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:56:52.179402 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:52.179371 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a-extensions-socket-volume\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:56:52.992760 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:52.992721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:52.995069 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:52.995031 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-c5mft\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:52.996716 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:52.996688 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:53.001374 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:53.001349 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-c5mft\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c5mft" Jul 20 02:56:53.002892 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:56:53.002870 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:56:53.194337 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:56:53.194270 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a" path="/var/lib/kubelet/pods/6fb9c87a-e3f4-4ab1-946c-4d47a0816e5a/volumes" Jul 20 02:57:03.230243 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.230218 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:57:03.231513 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:57:03.231480 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f45803_d9f7_4081_8aa5_98514d7bf978.slice/crio-77d762711d3bafd6d967bfbd4196d125230431d734211fd1684f4d9c716a296f WatchSource:0}: Error finding container 77d762711d3bafd6d967bfbd4196d125230431d734211fd1684f4d9c716a296f: Status 404 returned error can't find the container with id 77d762711d3bafd6d967bfbd4196d125230431d734211fd1684f4d9c716a296f Jul 20 02:57:03.237952 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.237933 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:03.239786 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:03.239763 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:03.255242 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.255220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 02:57:03.256841 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:57:03.256798 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1572dbee_2256_4071_9bf4_cec53ffb2ccd.slice/crio-e80a347ce8aa40cc207e7c131ddd1b70fec51b717befbf68e533961058d7405b WatchSource:0}: Error finding container e80a347ce8aa40cc207e7c131ddd1b70fec51b717befbf68e533961058d7405b: Status 404 returned error can't find the container with id e80a347ce8aa40cc207e7c131ddd1b70fec51b717befbf68e533961058d7405b Jul 20 02:57:03.390477 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.390444 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume\") pod \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " Jul 20 02:57:03.390671 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.390491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l55r9\" (UniqueName: \"kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9\") pod \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\" (UID: \"8f1b8f5b-e045-4b55-9991-70d6cf0ce719\") " Jul 20 02:57:03.390951 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.390926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8f1b8f5b-e045-4b55-9991-70d6cf0ce719" (UID: "8f1b8f5b-e045-4b55-9991-70d6cf0ce719"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:57:03.392514 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.392493 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9" (OuterVolumeSpecName: "kube-api-access-l55r9") pod "8f1b8f5b-e045-4b55-9991-70d6cf0ce719" (UID: "8f1b8f5b-e045-4b55-9991-70d6cf0ce719"). InnerVolumeSpecName "kube-api-access-l55r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:57:03.491250 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.491217 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-extensions-socket-volume\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:03.491250 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:03.491248 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55r9\" (UniqueName: \"kubernetes.io/projected/8f1b8f5b-e045-4b55-9991-70d6cf0ce719-kube-api-access-l55r9\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:04.062948 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.062909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" event={"ID":"1572dbee-2256-4071-9bf4-cec53ffb2ccd","Type":"ContainerStarted","Data":"e100807d478ee9eb7032d1b0ff9490fd7a314974dc093790394552199d9e2a71"} Jul 20 02:57:04.063163 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.062975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" event={"ID":"1572dbee-2256-4071-9bf4-cec53ffb2ccd","Type":"ContainerStarted","Data":"e80a347ce8aa40cc207e7c131ddd1b70fec51b717befbf68e533961058d7405b"} Jul 20 02:57:04.063163 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.063102 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:57:04.064674 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.064645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" event={"ID":"91f45803-d9f7-4081-8aa5-98514d7bf978","Type":"ContainerStarted","Data":"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c"} Jul 20 02:57:04.064800 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.064695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:57:04.064800 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.064720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" event={"ID":"91f45803-d9f7-4081-8aa5-98514d7bf978","Type":"ContainerStarted","Data":"77d762711d3bafd6d967bfbd4196d125230431d734211fd1684f4d9c716a296f"} Jul 20 02:57:04.066453 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.066430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" event={"ID":"26aa1c8f-e647-402d-8557-5cd153bfc98a","Type":"ContainerStarted","Data":"28e647a4fbf105f6fae0867a3e69b48f43d236eaa581edeb117d694c87b05c14"} Jul 20 02:57:04.067751 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.067730 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" containerID="c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7" exitCode=0 Jul 20 02:57:04.067835 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.067761 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:04.067835 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.067767 2565 scope.go:117] "RemoveContainer" containerID="c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7" Jul 20 02:57:04.080329 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:04.080263 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:04.081059 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.080983 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" podStartSLOduration=13.080963904 podStartE2EDuration="13.080963904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:04.078617058 +0000 UTC m=+718.954418768" watchObservedRunningTime="2026-07-20 02:57:04.080963904 +0000 UTC m=+718.956765615" Jul 20 02:57:04.081937 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:04.081905 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:04.086367 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.086346 2565 scope.go:117] "RemoveContainer" containerID="c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7" Jul 20 02:57:04.086689 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:04.086669 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7\": container with ID starting with c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7 not found: ID does not exist" containerID="c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7" Jul 20 02:57:04.086778 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.086700 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7"} err="failed to get container status \"c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7\": rpc error: code = NotFound desc = could not find container \"c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7\": container with ID starting with c17c6b949593a000541435e7185430d2719343a459aa857508bc1d6cf8ce73c7 not found: ID does not exist" Jul 20 02:57:04.096935 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.096894 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jhn8c" podStartSLOduration=2.48505023 podStartE2EDuration="23.096879265s" totalImagesPullingTime="20.611829035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:04.095842587 +0000 UTC m=+718.971644298" watchObservedRunningTime="2026-07-20 02:57:04.096879265 +0000 UTC m=+718.972680974" Jul 20 02:57:04.114703 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:04.114638 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" podStartSLOduration=13.11461611 podStartE2EDuration="13.11461611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:04.113209504 +0000 UTC m=+718.989011219" watchObservedRunningTime="2026-07-20 02:57:04.11461611 +0000 UTC m=+718.990417821" Jul 20 02:57:04.114967 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:04.114943 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-76g5c\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-76g5c" Jul 20 02:57:05.190510 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:05.190477 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1b8f5b-e045-4b55-9991-70d6cf0ce719" path="/var/lib/kubelet/pods/8f1b8f5b-e045-4b55-9991-70d6cf0ce719/volumes" Jul 20 02:57:15.075892 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:15.075859 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 02:57:15.076517 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:15.076059 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:57:15.146339 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:15.146307 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:57:15.146503 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:15.146479 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" podUID="91f45803-d9f7-4081-8aa5-98514d7bf978" containerName="manager" containerID="cri-o://179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c" gracePeriod=10 Jul 20 02:57:15.922337 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:15.922311 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:57:16.096781 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.096747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pftgs\" (UniqueName: \"kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs\") pod \"91f45803-d9f7-4081-8aa5-98514d7bf978\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " Jul 20 02:57:16.097251 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.096789 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume\") pod \"91f45803-d9f7-4081-8aa5-98514d7bf978\" (UID: \"91f45803-d9f7-4081-8aa5-98514d7bf978\") " Jul 20 02:57:16.097251 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.097167 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "91f45803-d9f7-4081-8aa5-98514d7bf978" (UID: "91f45803-d9f7-4081-8aa5-98514d7bf978"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:57:16.098900 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.098878 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs" (OuterVolumeSpecName: "kube-api-access-pftgs") pod "91f45803-d9f7-4081-8aa5-98514d7bf978" (UID: "91f45803-d9f7-4081-8aa5-98514d7bf978"). InnerVolumeSpecName "kube-api-access-pftgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:57:16.133863 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.133827 2565 generic.go:356] "Generic (PLEG): container finished" podID="91f45803-d9f7-4081-8aa5-98514d7bf978" containerID="179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c" exitCode=0 Jul 20 02:57:16.134049 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.133873 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" Jul 20 02:57:16.134049 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.133914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" event={"ID":"91f45803-d9f7-4081-8aa5-98514d7bf978","Type":"ContainerDied","Data":"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c"} Jul 20 02:57:16.134049 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.133968 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46" event={"ID":"91f45803-d9f7-4081-8aa5-98514d7bf978","Type":"ContainerDied","Data":"77d762711d3bafd6d967bfbd4196d125230431d734211fd1684f4d9c716a296f"} Jul 20 02:57:16.134049 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.133990 2565 scope.go:117] "RemoveContainer" containerID="179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c" Jul 20 02:57:16.143220 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.143203 2565 scope.go:117] "RemoveContainer" containerID="179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c" Jul 20 02:57:16.143530 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:16.143506 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c\": container with ID starting with 179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c not found: ID does not exist" containerID="179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c" Jul 20 02:57:16.143591 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.143545 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c"} err="failed to get container status \"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c\": rpc error: code = NotFound desc = could not find container \"179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c\": container with ID starting with 179dfc6739f94bb19e617b6da3ef68665d597bf88e25281b180f6a446801983c not found: ID does not exist" Jul 20 02:57:16.154360 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.154337 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:57:16.158037 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.158014 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nct46"] Jul 20 02:57:16.198345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.198315 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftgs\" (UniqueName: \"kubernetes.io/projected/91f45803-d9f7-4081-8aa5-98514d7bf978-kube-api-access-pftgs\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:16.198345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:16.198343 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91f45803-d9f7-4081-8aa5-98514d7bf978-extensions-socket-volume\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:17.191719 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:17.191681 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f45803-d9f7-4081-8aa5-98514d7bf978" path="/var/lib/kubelet/pods/91f45803-d9f7-4081-8aa5-98514d7bf978/volumes" Jul 20 02:57:33.208596 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.208563 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:33.208996 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.208891 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f45803-d9f7-4081-8aa5-98514d7bf978" containerName="manager" Jul 20 02:57:33.208996 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.208903 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f45803-d9f7-4081-8aa5-98514d7bf978" containerName="manager" Jul 20 02:57:33.208996 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.208972 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f45803-d9f7-4081-8aa5-98514d7bf978" containerName="manager" Jul 20 02:57:33.284259 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.284225 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:33.284439 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.284319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.286229 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.286203 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 02:57:33.298532 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.298507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:33.342644 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.342614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcpn\" (UniqueName: \"kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.342644 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.342648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.444106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.444070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcpn\" (UniqueName: \"kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.444106 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.444106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.444757 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.444731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.451272 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.451252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcpn\" (UniqueName: \"kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn\") pod \"limitador-limitador-64c94fd64f-64zqz\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.598772 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.598732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:33.747988 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.747964 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:33.749843 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:57:33.749816 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0e9486_bbaf_46e0_ae47_7361fc0e4e9f.slice/crio-698e70d0316e559cedfec3effb9bdfc2e5ca2f70d01976a2cb07742e4b0eccb1 WatchSource:0}: Error finding container 698e70d0316e559cedfec3effb9bdfc2e5ca2f70d01976a2cb07742e4b0eccb1: Status 404 returned error can't find the container with id 698e70d0316e559cedfec3effb9bdfc2e5ca2f70d01976a2cb07742e4b0eccb1 Jul 20 02:57:33.992709 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:33.992631 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:34.020814 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.020776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:34.020972 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.020877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:34.022800 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.022775 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wrzfl\"" Jul 20 02:57:34.149513 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.149480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nvf\" (UniqueName: \"kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf\") pod \"authorino-58cd955675-b2ptt\" (UID: \"baa3b4bd-6dac-4447-b510-eb15e765e51b\") " pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:34.196649 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.196611 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:57:34.247873 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.247797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:57:34.247873 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.247830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" event={"ID":"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f","Type":"ContainerStarted","Data":"698e70d0316e559cedfec3effb9bdfc2e5ca2f70d01976a2cb07742e4b0eccb1"} Jul 20 02:57:34.248263 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.247911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:57:34.250453 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.250422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nvf\" (UniqueName: \"kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf\") pod \"authorino-58cd955675-b2ptt\" (UID: \"baa3b4bd-6dac-4447-b510-eb15e765e51b\") " pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:34.264345 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.264316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nvf\" (UniqueName: \"kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf\") pod \"authorino-58cd955675-b2ptt\" (UID: \"baa3b4bd-6dac-4447-b510-eb15e765e51b\") " pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:34.331434 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.331391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:34.351730 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.351701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdjj\" (UniqueName: \"kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj\") pod \"authorino-75d556bd77-pnkqd\" (UID: \"ab78543e-b2ea-4035-b002-a29707d62a37\") " pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:57:34.453264 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.453225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdjj\" (UniqueName: \"kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj\") pod \"authorino-75d556bd77-pnkqd\" (UID: \"ab78543e-b2ea-4035-b002-a29707d62a37\") " pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:57:34.466469 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.466405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdjj\" (UniqueName: \"kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj\") pod \"authorino-75d556bd77-pnkqd\" (UID: \"ab78543e-b2ea-4035-b002-a29707d62a37\") " pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:57:34.492916 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.492889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:34.497436 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:57:34.497391 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaa3b4bd_6dac_4447_b510_eb15e765e51b.slice/crio-256968f5347aeb3dec2fac174d2d9383630b6d61bbeabee3d150aba441cae27f WatchSource:0}: Error finding container 256968f5347aeb3dec2fac174d2d9383630b6d61bbeabee3d150aba441cae27f: Status 404 returned error can't find the container with id 256968f5347aeb3dec2fac174d2d9383630b6d61bbeabee3d150aba441cae27f Jul 20 02:57:34.557320 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.557033 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:57:34.739615 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:34.739584 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:57:35.235841 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:35.235765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b2ptt" event={"ID":"baa3b4bd-6dac-4447-b510-eb15e765e51b","Type":"ContainerStarted","Data":"256968f5347aeb3dec2fac174d2d9383630b6d61bbeabee3d150aba441cae27f"} Jul 20 02:57:35.241160 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:35.241068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pnkqd" event={"ID":"ab78543e-b2ea-4035-b002-a29707d62a37","Type":"ContainerStarted","Data":"73b8d417cd362e0fb0f38895a7166e3539c901ff6866d7ca428fe8871260f952"} Jul 20 02:57:39.280926 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.280735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" event={"ID":"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f","Type":"ContainerStarted","Data":"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e"} Jul 20 02:57:39.281438 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.281048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:39.283981 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.283951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b2ptt" event={"ID":"baa3b4bd-6dac-4447-b510-eb15e765e51b","Type":"ContainerStarted","Data":"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0"} Jul 20 02:57:39.286084 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.286055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pnkqd" event={"ID":"ab78543e-b2ea-4035-b002-a29707d62a37","Type":"ContainerStarted","Data":"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357"} Jul 20 02:57:39.296845 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.296788 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" podStartSLOduration=1.077890722 podStartE2EDuration="6.29677476s" totalImagesPullingTime="5.218884038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:39.295702354 +0000 UTC m=+754.171504065" watchObservedRunningTime="2026-07-20 02:57:39.29677476 +0000 UTC m=+754.172576469" Jul 20 02:57:39.313569 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.313528 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-b2ptt" podStartSLOduration=1.829602724 podStartE2EDuration="6.313515327s" totalImagesPullingTime="4.483912603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:39.312087806 +0000 UTC m=+754.187889516" watchObservedRunningTime="2026-07-20 02:57:39.313515327 +0000 UTC m=+754.189317036" Jul 20 02:57:39.326055 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.326001 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pnkqd" podStartSLOduration=1.09348314 podStartE2EDuration="5.325983544s" totalImagesPullingTime="4.232500404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:39.323433011 +0000 UTC m=+754.199234722" watchObservedRunningTime="2026-07-20 02:57:39.325983544 +0000 UTC m=+754.201785256" Jul 20 02:57:39.348571 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:39.348542 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:41.296921 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:41.296875 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-b2ptt" podUID="baa3b4bd-6dac-4447-b510-eb15e765e51b" containerName="authorino" containerID="cri-o://933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0" gracePeriod=30 Jul 20 02:57:41.540308 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:41.540274 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:41.624413 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:41.624381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8nvf\" (UniqueName: \"kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf\") pod \"baa3b4bd-6dac-4447-b510-eb15e765e51b\" (UID: \"baa3b4bd-6dac-4447-b510-eb15e765e51b\") " Jul 20 02:57:41.626463 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:41.626434 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf" (OuterVolumeSpecName: "kube-api-access-p8nvf") pod "baa3b4bd-6dac-4447-b510-eb15e765e51b" (UID: "baa3b4bd-6dac-4447-b510-eb15e765e51b"). InnerVolumeSpecName "kube-api-access-p8nvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:57:41.726018 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:41.725984 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nvf\" (UniqueName: \"kubernetes.io/projected/baa3b4bd-6dac-4447-b510-eb15e765e51b-kube-api-access-p8nvf\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:42.302911 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.302875 2565 generic.go:356] "Generic (PLEG): container finished" podID="baa3b4bd-6dac-4447-b510-eb15e765e51b" containerID="933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0" exitCode=0 Jul 20 02:57:42.303344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.302924 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b2ptt" Jul 20 02:57:42.303344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.302965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b2ptt" event={"ID":"baa3b4bd-6dac-4447-b510-eb15e765e51b","Type":"ContainerDied","Data":"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0"} Jul 20 02:57:42.303344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.303012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b2ptt" event={"ID":"baa3b4bd-6dac-4447-b510-eb15e765e51b","Type":"ContainerDied","Data":"256968f5347aeb3dec2fac174d2d9383630b6d61bbeabee3d150aba441cae27f"} Jul 20 02:57:42.303344 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.303033 2565 scope.go:117] "RemoveContainer" containerID="933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0" Jul 20 02:57:42.312254 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.312236 2565 scope.go:117] "RemoveContainer" containerID="933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0" Jul 20 02:57:42.312548 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:42.312526 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0\": container with ID starting with 933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0 not found: ID does not exist" containerID="933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0" Jul 20 02:57:42.312605 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.312555 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0"} err="failed to get container status \"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0\": rpc error: code = NotFound desc = could not find container \"933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0\": container with ID starting with 933fac7a08686e9a14fafaf084f110f61d8b05eea833690f1621243ec88a00e0 not found: ID does not exist" Jul 20 02:57:42.321568 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.321543 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:42.324626 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:42.324605 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-b2ptt"] Jul 20 02:57:43.190400 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:43.190366 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa3b4bd-6dac-4447-b510-eb15e765e51b" path="/var/lib/kubelet/pods/baa3b4bd-6dac-4447-b510-eb15e765e51b/volumes" Jul 20 02:57:48.292660 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.292626 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:48.293165 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.292851 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" podUID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" containerName="limitador" containerID="cri-o://0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e" gracePeriod=30 Jul 20 02:57:48.295026 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.295002 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:48.853090 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.853038 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:48.992551 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.992519 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btcpn\" (UniqueName: \"kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn\") pod \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " Jul 20 02:57:48.992735 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.992619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file\") pod \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\" (UID: \"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f\") " Jul 20 02:57:48.992964 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.992937 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file" (OuterVolumeSpecName: "config-file") pod "8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" (UID: "8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:57:48.994602 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:48.994581 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn" (OuterVolumeSpecName: "kube-api-access-btcpn") pod "8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" (UID: "8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f"). InnerVolumeSpecName "kube-api-access-btcpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:57:49.093885 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.093850 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-config-file\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:49.093885 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.093880 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcpn\" (UniqueName: \"kubernetes.io/projected/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f-kube-api-access-btcpn\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:57:49.344127 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.344093 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" containerID="0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e" exitCode=0 Jul 20 02:57:49.344607 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.344138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" Jul 20 02:57:49.344607 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.344158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" event={"ID":"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f","Type":"ContainerDied","Data":"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e"} Jul 20 02:57:49.344607 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.344204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-64zqz" event={"ID":"8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f","Type":"ContainerDied","Data":"698e70d0316e559cedfec3effb9bdfc2e5ca2f70d01976a2cb07742e4b0eccb1"} Jul 20 02:57:49.344607 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.344226 2565 scope.go:117] "RemoveContainer" containerID="0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e" Jul 20 02:57:49.352928 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.352880 2565 scope.go:117] "RemoveContainer" containerID="0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e" Jul 20 02:57:49.353164 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:57:49.353146 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e\": container with ID starting with 0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e not found: ID does not exist" containerID="0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e" Jul 20 02:57:49.353240 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.353172 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e"} err="failed to get container status \"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e\": rpc error: code = NotFound desc = could not find container \"0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e\": container with ID starting with 0f4bcb9c4ffb5abb90a18438da254c6b76fc10831c9b4813c3b3245ce6416c6e not found: ID does not exist" Jul 20 02:57:49.367470 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.367442 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:49.369338 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.369320 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-64zqz"] Jul 20 02:57:49.858025 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.857991 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-td7k5"] Jul 20 02:57:49.858426 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.858409 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" containerName="limitador" Jul 20 02:57:49.858557 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.858429 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" containerName="limitador" Jul 20 02:57:49.858557 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.858446 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa3b4bd-6dac-4447-b510-eb15e765e51b" containerName="authorino" Jul 20 02:57:49.858557 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.858455 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa3b4bd-6dac-4447-b510-eb15e765e51b" containerName="authorino" Jul 20 02:57:49.859069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.859043 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" containerName="limitador" Jul 20 02:57:49.859069 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.859071 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa3b4bd-6dac-4447-b510-eb15e765e51b" containerName="authorino" Jul 20 02:57:49.869519 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.869493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:49.871270 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.871206 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-td7k5"] Jul 20 02:57:49.871930 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.871909 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 02:57:49.872172 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.872155 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 20 02:57:49.872483 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.872467 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 02:57:49.872669 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:49.872656 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xmt4s\"" Jul 20 02:57:50.001446 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.001411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8r8k\" (UniqueName: \"kubernetes.io/projected/aa076322-2904-4691-93ba-20979ff3ea6f-kube-api-access-w8r8k\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.001611 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.001485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa076322-2904-4691-93ba-20979ff3ea6f-data\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.102915 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.102883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8r8k\" (UniqueName: \"kubernetes.io/projected/aa076322-2904-4691-93ba-20979ff3ea6f-kube-api-access-w8r8k\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.103068 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.102962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa076322-2904-4691-93ba-20979ff3ea6f-data\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.103391 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.103372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa076322-2904-4691-93ba-20979ff3ea6f-data\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.110719 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.110661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8r8k\" (UniqueName: \"kubernetes.io/projected/aa076322-2904-4691-93ba-20979ff3ea6f-kube-api-access-w8r8k\") pod \"postgres-69fb4594cf-td7k5\" (UID: \"aa076322-2904-4691-93ba-20979ff3ea6f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.181672 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.181639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:50.311517 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.311488 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-td7k5"] Jul 20 02:57:50.313395 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:57:50.313357 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa076322_2904_4691_93ba_20979ff3ea6f.slice/crio-c8cfd56d218234d2510782b17187edebc3abb2d34876678788df145a9f18520b WatchSource:0}: Error finding container c8cfd56d218234d2510782b17187edebc3abb2d34876678788df145a9f18520b: Status 404 returned error can't find the container with id c8cfd56d218234d2510782b17187edebc3abb2d34876678788df145a9f18520b Jul 20 02:57:50.350530 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:50.350496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" event={"ID":"aa076322-2904-4691-93ba-20979ff3ea6f","Type":"ContainerStarted","Data":"c8cfd56d218234d2510782b17187edebc3abb2d34876678788df145a9f18520b"} Jul 20 02:57:51.193595 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:51.193563 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f" path="/var/lib/kubelet/pods/8f0e9486-bbaf-46e0-ae47-7361fc0e4e9f/volumes" Jul 20 02:57:55.384437 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:55.384405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" event={"ID":"aa076322-2904-4691-93ba-20979ff3ea6f","Type":"ContainerStarted","Data":"6ed4e3d73ddba188887b358280a69cfc2162bc31563ee26fb1e55e3eba1ed6eb"} Jul 20 02:57:55.384832 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:55.384522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:57:55.398180 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:57:55.398009 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" podStartSLOduration=1.501253219 podStartE2EDuration="6.397992049s" totalImagesPullingTime="4.89673883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:55.397584174 +0000 UTC m=+770.273385884" watchObservedRunningTime="2026-07-20 02:57:55.397992049 +0000 UTC m=+770.273793760" Jul 20 02:58:01.420350 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:01.420305 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-td7k5" Jul 20 02:58:01.949965 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:01.949927 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:01.953281 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:01.953256 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:01.958755 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:01.958729 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:02.007096 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.007058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnnf\" (UniqueName: \"kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf\") pod \"authorino-5d4b67c446-msf7f\" (UID: \"c0b68578-c87d-4b87-982a-8095db127121\") " pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:02.108236 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.108197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnnf\" (UniqueName: \"kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf\") pod \"authorino-5d4b67c446-msf7f\" (UID: \"c0b68578-c87d-4b87-982a-8095db127121\") " pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:02.115556 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.115531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnnf\" (UniqueName: \"kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf\") pod \"authorino-5d4b67c446-msf7f\" (UID: \"c0b68578-c87d-4b87-982a-8095db127121\") " pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:02.197348 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.197315 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:02.197575 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.197564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:02.220313 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.220263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-z7b67"] Jul 20 02:58:02.223797 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.223772 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.225885 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.225863 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 02:58:02.239331 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.235421 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-z7b67"] Jul 20 02:58:02.250954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.250926 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-z7b67"] Jul 20 02:58:02.251231 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:02.251200 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hxfv5 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-hxfv5 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-z7b67" podUID="119fd170-2c23-4406-8021-989b78690f5a" Jul 20 02:58:02.277586 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.277558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 02:58:02.280857 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.280838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.289640 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.289586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 02:58:02.309933 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.309898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.310178 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.310156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf84x\" (UniqueName: \"kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.335769 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.335747 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:02.337559 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:02.337535 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0b68578_c87d_4b87_982a_8095db127121.slice/crio-e8ecd6633e4b57c3558d63c62dbee50c61c159ead2cfdd55faa5bf18693c1f43 WatchSource:0}: Error finding container e8ecd6633e4b57c3558d63c62dbee50c61c159ead2cfdd55faa5bf18693c1f43: Status 404 returned error can't find the container with id e8ecd6633e4b57c3558d63c62dbee50c61c159ead2cfdd55faa5bf18693c1f43 Jul 20 02:58:02.411508 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.411474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfv5\" (UniqueName: \"kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.411682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.411552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.411682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.411584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf84x\" (UniqueName: \"kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.411682 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.411611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.414165 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.414134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.420047 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.420019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf84x\" (UniqueName: \"kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x\") pod \"authorino-5655c9f75c-bbq8k\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.423652 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.423627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-msf7f" event={"ID":"c0b68578-c87d-4b87-982a-8095db127121","Type":"ContainerStarted","Data":"e8ecd6633e4b57c3558d63c62dbee50c61c159ead2cfdd55faa5bf18693c1f43"} Jul 20 02:58:02.423961 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.423663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.430460 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.430441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.512676 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.512595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.512676 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.512645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfv5\" (UniqueName: \"kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.515006 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.514985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.520389 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.520370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfv5\" (UniqueName: \"kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5\") pod \"authorino-76877994-z7b67\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:02.592145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.592114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 02:58:02.613174 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.613142 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxfv5\" (UniqueName: \"kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5\") pod \"119fd170-2c23-4406-8021-989b78690f5a\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " Jul 20 02:58:02.613341 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.613205 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert\") pod \"119fd170-2c23-4406-8021-989b78690f5a\" (UID: \"119fd170-2c23-4406-8021-989b78690f5a\") " Jul 20 02:58:02.615354 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.615321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "119fd170-2c23-4406-8021-989b78690f5a" (UID: "119fd170-2c23-4406-8021-989b78690f5a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:58:02.615468 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.615321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5" (OuterVolumeSpecName: "kube-api-access-hxfv5") pod "119fd170-2c23-4406-8021-989b78690f5a" (UID: "119fd170-2c23-4406-8021-989b78690f5a"). InnerVolumeSpecName "kube-api-access-hxfv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:58:02.714053 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.714027 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfv5\" (UniqueName: \"kubernetes.io/projected/119fd170-2c23-4406-8021-989b78690f5a-kube-api-access-hxfv5\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:02.714053 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.714052 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/119fd170-2c23-4406-8021-989b78690f5a-tls-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:02.723552 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:02.723529 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 02:58:02.725670 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:02.725645 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22e16fca_d5f5_4714_9221_95561d750aaa.slice/crio-8b00c747e97d80fad0fef53521f22e984771dcad1b1b8ac0a24fa8f56536b765 WatchSource:0}: Error finding container 8b00c747e97d80fad0fef53521f22e984771dcad1b1b8ac0a24fa8f56536b765: Status 404 returned error can't find the container with id 8b00c747e97d80fad0fef53521f22e984771dcad1b1b8ac0a24fa8f56536b765 Jul 20 02:58:03.430330 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.430273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-msf7f" event={"ID":"c0b68578-c87d-4b87-982a-8095db127121","Type":"ContainerStarted","Data":"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df"} Jul 20 02:58:03.430330 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.430285 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-msf7f" podUID="c0b68578-c87d-4b87-982a-8095db127121" containerName="authorino" containerID="cri-o://eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df" gracePeriod=30 Jul 20 02:58:03.431954 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.431925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" event={"ID":"22e16fca-d5f5-4714-9221-95561d750aaa","Type":"ContainerStarted","Data":"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb"} Jul 20 02:58:03.432095 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.431966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-z7b67" Jul 20 02:58:03.432095 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.431966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" event={"ID":"22e16fca-d5f5-4714-9221-95561d750aaa","Type":"ContainerStarted","Data":"8b00c747e97d80fad0fef53521f22e984771dcad1b1b8ac0a24fa8f56536b765"} Jul 20 02:58:03.445558 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.445510 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-msf7f" podStartSLOduration=2.128377681 podStartE2EDuration="2.44549148s" totalImagesPullingTime="317.113799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:03.443131014 +0000 UTC m=+778.318932726" watchObservedRunningTime="2026-07-20 02:58:03.44549148 +0000 UTC m=+778.321293192" Jul 20 02:58:03.464981 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.464914 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-z7b67"] Jul 20 02:58:03.468250 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.468224 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-z7b67"] Jul 20 02:58:03.480016 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.479972 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" podStartSLOduration=0.962910083 podStartE2EDuration="1.479957999s" totalImagesPullingTime="517.047916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:03.47847878 +0000 UTC m=+778.354280490" watchObservedRunningTime="2026-07-20 02:58:03.479957999 +0000 UTC m=+778.355759710" Jul 20 02:58:03.502244 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.502213 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:58:03.502414 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.502391 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pnkqd" podUID="ab78543e-b2ea-4035-b002-a29707d62a37" containerName="authorino" containerID="cri-o://423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357" gracePeriod=30 Jul 20 02:58:03.761541 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.761518 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:03.766085 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.766064 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:58:03.823542 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.823512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmdjj\" (UniqueName: \"kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj\") pod \"ab78543e-b2ea-4035-b002-a29707d62a37\" (UID: \"ab78543e-b2ea-4035-b002-a29707d62a37\") " Jul 20 02:58:03.823693 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.823558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtnnf\" (UniqueName: \"kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf\") pod \"c0b68578-c87d-4b87-982a-8095db127121\" (UID: \"c0b68578-c87d-4b87-982a-8095db127121\") " Jul 20 02:58:03.825651 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.825605 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj" (OuterVolumeSpecName: "kube-api-access-nmdjj") pod "ab78543e-b2ea-4035-b002-a29707d62a37" (UID: "ab78543e-b2ea-4035-b002-a29707d62a37"). InnerVolumeSpecName "kube-api-access-nmdjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:58:03.825651 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.825633 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf" (OuterVolumeSpecName: "kube-api-access-dtnnf") pod "c0b68578-c87d-4b87-982a-8095db127121" (UID: "c0b68578-c87d-4b87-982a-8095db127121"). InnerVolumeSpecName "kube-api-access-dtnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:58:03.924353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.924320 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdjj\" (UniqueName: \"kubernetes.io/projected/ab78543e-b2ea-4035-b002-a29707d62a37-kube-api-access-nmdjj\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:03.924353 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:03.924348 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtnnf\" (UniqueName: \"kubernetes.io/projected/c0b68578-c87d-4b87-982a-8095db127121-kube-api-access-dtnnf\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:04.438123 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.438047 2565 generic.go:356] "Generic (PLEG): container finished" podID="ab78543e-b2ea-4035-b002-a29707d62a37" containerID="423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357" exitCode=0 Jul 20 02:58:04.438123 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.438090 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pnkqd" Jul 20 02:58:04.438598 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.438134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pnkqd" event={"ID":"ab78543e-b2ea-4035-b002-a29707d62a37","Type":"ContainerDied","Data":"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357"} Jul 20 02:58:04.438598 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.438172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pnkqd" event={"ID":"ab78543e-b2ea-4035-b002-a29707d62a37","Type":"ContainerDied","Data":"73b8d417cd362e0fb0f38895a7166e3539c901ff6866d7ca428fe8871260f952"} Jul 20 02:58:04.438598 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.438196 2565 scope.go:117] "RemoveContainer" containerID="423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357" Jul 20 02:58:04.439570 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.439545 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b68578-c87d-4b87-982a-8095db127121" containerID="eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df" exitCode=0 Jul 20 02:58:04.439648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.439578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-msf7f" event={"ID":"c0b68578-c87d-4b87-982a-8095db127121","Type":"ContainerDied","Data":"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df"} Jul 20 02:58:04.439648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.439607 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-msf7f" Jul 20 02:58:04.439648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.439624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-msf7f" event={"ID":"c0b68578-c87d-4b87-982a-8095db127121","Type":"ContainerDied","Data":"e8ecd6633e4b57c3558d63c62dbee50c61c159ead2cfdd55faa5bf18693c1f43"} Jul 20 02:58:04.449137 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.449088 2565 scope.go:117] "RemoveContainer" containerID="423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357" Jul 20 02:58:04.449407 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:04.449389 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357\": container with ID starting with 423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357 not found: ID does not exist" containerID="423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357" Jul 20 02:58:04.449480 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.449416 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357"} err="failed to get container status \"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357\": rpc error: code = NotFound desc = could not find container \"423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357\": container with ID starting with 423946c04b9ff047d89c29ae2edb06f9d65f086a11dcb6109d4a99d35b1d2357 not found: ID does not exist" Jul 20 02:58:04.449480 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.449435 2565 scope.go:117] "RemoveContainer" containerID="eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df" Jul 20 02:58:04.458002 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.457962 2565 scope.go:117] "RemoveContainer" containerID="eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df" Jul 20 02:58:04.458257 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:04.458232 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df\": container with ID starting with eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df not found: ID does not exist" containerID="eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df" Jul 20 02:58:04.458425 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.458266 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df"} err="failed to get container status \"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df\": rpc error: code = NotFound desc = could not find container \"eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df\": container with ID starting with eb26f14c448d6104f90b859b20bfade02ff6ca7fda31c7619280d6c6a69248df not found: ID does not exist" Jul 20 02:58:04.459206 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.459185 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:58:04.461203 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.461182 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pnkqd"] Jul 20 02:58:04.468813 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.468787 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:04.470782 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:04.470763 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-msf7f"] Jul 20 02:58:05.191847 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:05.191812 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119fd170-2c23-4406-8021-989b78690f5a" path="/var/lib/kubelet/pods/119fd170-2c23-4406-8021-989b78690f5a/volumes" Jul 20 02:58:05.192193 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:05.192174 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab78543e-b2ea-4035-b002-a29707d62a37" path="/var/lib/kubelet/pods/ab78543e-b2ea-4035-b002-a29707d62a37/volumes" Jul 20 02:58:05.192593 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:05.192579 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b68578-c87d-4b87-982a-8095db127121" path="/var/lib/kubelet/pods/c0b68578-c87d-4b87-982a-8095db127121/volumes" Jul 20 02:58:08.280590 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280554 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:08.280969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280894 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab78543e-b2ea-4035-b002-a29707d62a37" containerName="authorino" Jul 20 02:58:08.280969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280906 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab78543e-b2ea-4035-b002-a29707d62a37" containerName="authorino" Jul 20 02:58:08.280969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280925 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b68578-c87d-4b87-982a-8095db127121" containerName="authorino" Jul 20 02:58:08.280969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280931 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b68578-c87d-4b87-982a-8095db127121" containerName="authorino" Jul 20 02:58:08.281093 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280989 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab78543e-b2ea-4035-b002-a29707d62a37" containerName="authorino" Jul 20 02:58:08.281093 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.280999 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b68578-c87d-4b87-982a-8095db127121" containerName="authorino" Jul 20 02:58:08.404949 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.404909 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:08.405161 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.405023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.407242 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.407212 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 02:58:08.407378 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.407326 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb24c\"" Jul 20 02:58:08.407378 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.407326 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 02:58:08.458477 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.458443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf86\" (UniqueName: \"kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.458615 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.458533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.491537 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.491502 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 02:58:08.522269 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.522235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 02:58:08.522455 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.522379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:08.559201 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.559133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:08.559201 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.559184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf86\" (UniqueName: \"kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.559395 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.559220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwrt\" (UniqueName: \"kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:08.559395 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.559366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.559513 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:08.559497 2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 20 02:58:08.559563 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:08.559554 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert podName:392b5cb3-0c17-4e45-bec4-d0b352bd87b1 nodeName:}" failed. No retries permitted until 2026-07-20 02:58:09.059538095 +0000 UTC m=+783.935339786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert") pod "maas-controller-6dcfbb5dd5-z4jcg" (UID: "392b5cb3-0c17-4e45-bec4-d0b352bd87b1") : secret "maas-controller-webhook-cert" not found Jul 20 02:58:08.568746 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.568721 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf86\" (UniqueName: \"kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:08.660553 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.660521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:08.660688 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.660569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwrt\" (UniqueName: \"kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:08.660742 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:08.660681 2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 20 02:58:08.660774 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:08.660745 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert podName:5e54b88a-49ab-4888-a5ac-6c6e97f04aea nodeName:}" failed. No retries permitted until 2026-07-20 02:58:09.160727749 +0000 UTC m=+784.036529437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert") pod "maas-controller-645d8c8bc6-k4tgk" (UID: "5e54b88a-49ab-4888-a5ac-6c6e97f04aea") : secret "maas-controller-webhook-cert" not found Jul 20 02:58:08.670477 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:08.670448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwrt\" (UniqueName: \"kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:09.063170 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:09.063133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:09.063382 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:09.063251 2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 20 02:58:09.063382 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:09.063325 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert podName:392b5cb3-0c17-4e45-bec4-d0b352bd87b1 nodeName:}" failed. No retries permitted until 2026-07-20 02:58:10.063309923 +0000 UTC m=+784.939111624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert") pod "maas-controller-6dcfbb5dd5-z4jcg" (UID: "392b5cb3-0c17-4e45-bec4-d0b352bd87b1") : secret "maas-controller-webhook-cert" not found Jul 20 02:58:09.163720 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:09.163673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:09.165976 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:09.165955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") pod \"maas-controller-645d8c8bc6-k4tgk\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:09.432966 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:09.432930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:09.610472 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:09.610442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 02:58:09.611769 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:09.611742 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e54b88a_49ab_4888_a5ac_6c6e97f04aea.slice/crio-a0cafb9f80b2da889d6775bad8a868ae9e9347cd97174cc1b3cfb73d304a1d58 WatchSource:0}: Error finding container a0cafb9f80b2da889d6775bad8a868ae9e9347cd97174cc1b3cfb73d304a1d58: Status 404 returned error can't find the container with id a0cafb9f80b2da889d6775bad8a868ae9e9347cd97174cc1b3cfb73d304a1d58 Jul 20 02:58:10.071317 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.071262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:10.073576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.073549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"maas-controller-6dcfbb5dd5-z4jcg\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:10.215360 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.215319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:10.389160 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.388870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:10.392457 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:10.392285 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392b5cb3_0c17_4e45_bec4_d0b352bd87b1.slice/crio-e58bfba478ffaf3e882c0683ac30814eb0bcca57c3b109ac588fc9c8fc34b129 WatchSource:0}: Error finding container e58bfba478ffaf3e882c0683ac30814eb0bcca57c3b109ac588fc9c8fc34b129: Status 404 returned error can't find the container with id e58bfba478ffaf3e882c0683ac30814eb0bcca57c3b109ac588fc9c8fc34b129 Jul 20 02:58:10.478493 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.478456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" event={"ID":"392b5cb3-0c17-4e45-bec4-d0b352bd87b1","Type":"ContainerStarted","Data":"e58bfba478ffaf3e882c0683ac30814eb0bcca57c3b109ac588fc9c8fc34b129"} Jul 20 02:58:10.480123 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:10.480090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" event={"ID":"5e54b88a-49ab-4888-a5ac-6c6e97f04aea","Type":"ContainerStarted","Data":"a0cafb9f80b2da889d6775bad8a868ae9e9347cd97174cc1b3cfb73d304a1d58"} Jul 20 02:58:12.493791 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.493757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" event={"ID":"5e54b88a-49ab-4888-a5ac-6c6e97f04aea","Type":"ContainerStarted","Data":"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18"} Jul 20 02:58:12.494285 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.493931 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:12.495366 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.495341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" event={"ID":"392b5cb3-0c17-4e45-bec4-d0b352bd87b1","Type":"ContainerStarted","Data":"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903"} Jul 20 02:58:12.495482 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.495468 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:12.520521 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.520464 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" podStartSLOduration=2.061261722 podStartE2EDuration="4.520451488s" totalImagesPullingTime="2.459189766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:12.518528617 +0000 UTC m=+787.394330328" watchObservedRunningTime="2026-07-20 02:58:12.520451488 +0000 UTC m=+787.396253197" Jul 20 02:58:12.535461 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:12.535401 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" podStartSLOduration=2.847418733 podStartE2EDuration="4.535386369s" totalImagesPullingTime="1.687967636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:12.534186652 +0000 UTC m=+787.409988363" watchObservedRunningTime="2026-07-20 02:58:12.535386369 +0000 UTC m=+787.411188079" Jul 20 02:58:23.502757 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.502719 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:23.503145 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.502885 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 02:58:23.556615 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.556582 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:23.557078 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.557040 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" podUID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" containerName="manager" containerID="cri-o://7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903" gracePeriod=30 Jul 20 02:58:23.808746 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.808721 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:23.898961 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.898929 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") pod \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " Jul 20 02:58:23.899155 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.899029 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxf86\" (UniqueName: \"kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86\") pod \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\" (UID: \"392b5cb3-0c17-4e45-bec4-d0b352bd87b1\") " Jul 20 02:58:23.901197 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.901161 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86" (OuterVolumeSpecName: "kube-api-access-dxf86") pod "392b5cb3-0c17-4e45-bec4-d0b352bd87b1" (UID: "392b5cb3-0c17-4e45-bec4-d0b352bd87b1"). InnerVolumeSpecName "kube-api-access-dxf86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:58:23.901327 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:23.901173 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "392b5cb3-0c17-4e45-bec4-d0b352bd87b1" (UID: "392b5cb3-0c17-4e45-bec4-d0b352bd87b1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:58:24.000337 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.000275 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-webhook-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:24.000337 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.000331 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf86\" (UniqueName: \"kubernetes.io/projected/392b5cb3-0c17-4e45-bec4-d0b352bd87b1-kube-api-access-dxf86\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 02:58:24.560363 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.560323 2565 generic.go:356] "Generic (PLEG): container finished" podID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" containerID="7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903" exitCode=0 Jul 20 02:58:24.560363 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.560365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" Jul 20 02:58:24.560876 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.560411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" event={"ID":"392b5cb3-0c17-4e45-bec4-d0b352bd87b1","Type":"ContainerDied","Data":"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903"} Jul 20 02:58:24.560876 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.560474 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcfbb5dd5-z4jcg" event={"ID":"392b5cb3-0c17-4e45-bec4-d0b352bd87b1","Type":"ContainerDied","Data":"e58bfba478ffaf3e882c0683ac30814eb0bcca57c3b109ac588fc9c8fc34b129"} Jul 20 02:58:24.560876 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.560512 2565 scope.go:117] "RemoveContainer" containerID="7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903" Jul 20 02:58:24.569563 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.569547 2565 scope.go:117] "RemoveContainer" containerID="7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903" Jul 20 02:58:24.569825 ip-10-0-134-38 kubenswrapper[2565]: E0720 02:58:24.569805 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903\": container with ID starting with 7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903 not found: ID does not exist" containerID="7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903" Jul 20 02:58:24.569869 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.569834 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903"} err="failed to get container status \"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903\": rpc error: code = NotFound desc = could not find container \"7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903\": container with ID starting with 7d283896b28cc82fef126ef279a91615a8ff67e7383629ddf6ea349165887903 not found: ID does not exist" Jul 20 02:58:24.581125 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.581097 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:24.584442 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:24.584408 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dcfbb5dd5-z4jcg"] Jul 20 02:58:25.190870 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:25.190838 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" path="/var/lib/kubelet/pods/392b5cb3-0c17-4e45-bec4-d0b352bd87b1/volumes" Jul 20 02:58:44.424354 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.424322 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd"] Jul 20 02:58:44.424951 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.424862 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" containerName="manager" Jul 20 02:58:44.424951 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.424879 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" containerName="manager" Jul 20 02:58:44.424951 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.424945 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="392b5cb3-0c17-4e45-bec4-d0b352bd87b1" containerName="manager" Jul 20 02:58:44.433557 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.433533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.436092 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.436067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r4rwt\"" Jul 20 02:58:44.436232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.436079 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 20 02:58:44.438384 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.438358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd"] Jul 20 02:58:44.582912 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.582877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59ddee8-502d-4ee4-b037-d3b09a18062a-maas-api-tls\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.583086 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.582956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59ddee8-502d-4ee4-b037-d3b09a18062a-openshift-service-ca\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.583086 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.582978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh6n\" (UniqueName: \"kubernetes.io/projected/c59ddee8-502d-4ee4-b037-d3b09a18062a-kube-api-access-vxh6n\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.684008 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.683920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59ddee8-502d-4ee4-b037-d3b09a18062a-maas-api-tls\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.684173 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.684012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59ddee8-502d-4ee4-b037-d3b09a18062a-openshift-service-ca\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.684173 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.684040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh6n\" (UniqueName: \"kubernetes.io/projected/c59ddee8-502d-4ee4-b037-d3b09a18062a-kube-api-access-vxh6n\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.684662 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.684642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59ddee8-502d-4ee4-b037-d3b09a18062a-openshift-service-ca\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.686323 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.686302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59ddee8-502d-4ee4-b037-d3b09a18062a-maas-api-tls\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.692224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.692192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh6n\" (UniqueName: \"kubernetes.io/projected/c59ddee8-502d-4ee4-b037-d3b09a18062a-kube-api-access-vxh6n\") pod \"maas-api-79b94c59c6-fzswd\" (UID: \"c59ddee8-502d-4ee4-b037-d3b09a18062a\") " pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.745654 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.745625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:44.883519 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:44.883492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd"] Jul 20 02:58:44.884870 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:44.884844 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59ddee8_502d_4ee4_b037_d3b09a18062a.slice/crio-0830b03a12e164dd2a1d967d52b56ead489afd070d43f1aff15ee441cd51303f WatchSource:0}: Error finding container 0830b03a12e164dd2a1d967d52b56ead489afd070d43f1aff15ee441cd51303f: Status 404 returned error can't find the container with id 0830b03a12e164dd2a1d967d52b56ead489afd070d43f1aff15ee441cd51303f Jul 20 02:58:45.079150 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.079122 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bltbd"] Jul 20 02:58:45.084268 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.084252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.086617 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.086596 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-w2cpp\"" Jul 20 02:58:45.086812 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.086798 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 20 02:58:45.093397 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.093367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bltbd"] Jul 20 02:58:45.112376 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.112349 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hk2g6"] Jul 20 02:58:45.116063 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.116047 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.126264 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.126237 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hk2g6"] Jul 20 02:58:45.187896 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.187864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f000f6-0a19-4e6c-aded-34677ef8fc3d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.188085 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.187920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws75\" (UniqueName: \"kubernetes.io/projected/65f000f6-0a19-4e6c-aded-34677ef8fc3d-kube-api-access-zws75\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.288986 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.288942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpb9\" (UniqueName: \"kubernetes.io/projected/773f6e47-f1d3-4555-8680-c23e0ce263c3-kube-api-access-5fpb9\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.289181 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.289032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f000f6-0a19-4e6c-aded-34677ef8fc3d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.289181 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.289070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zws75\" (UniqueName: \"kubernetes.io/projected/65f000f6-0a19-4e6c-aded-34677ef8fc3d-kube-api-access-zws75\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.289181 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.289109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773f6e47-f1d3-4555-8680-c23e0ce263c3-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.289969 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.289943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f000f6-0a19-4e6c-aded-34677ef8fc3d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.297044 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.297023 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws75\" (UniqueName: \"kubernetes.io/projected/65f000f6-0a19-4e6c-aded-34677ef8fc3d-kube-api-access-zws75\") pod \"payload-pre-processing-7677b774bb-bltbd\" (UID: \"65f000f6-0a19-4e6c-aded-34677ef8fc3d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.390352 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.390263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpb9\" (UniqueName: \"kubernetes.io/projected/773f6e47-f1d3-4555-8680-c23e0ce263c3-kube-api-access-5fpb9\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.390526 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.390405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773f6e47-f1d3-4555-8680-c23e0ce263c3-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.391167 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.391143 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773f6e47-f1d3-4555-8680-c23e0ce263c3-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.396154 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.396131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:45.398389 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.398367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpb9\" (UniqueName: \"kubernetes.io/projected/773f6e47-f1d3-4555-8680-c23e0ce263c3-kube-api-access-5fpb9\") pod \"payload-processing-5bf47cd988-hk2g6\" (UID: \"773f6e47-f1d3-4555-8680-c23e0ce263c3\") " pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.428664 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.428637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:45.669021 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.668931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" event={"ID":"c59ddee8-502d-4ee4-b037-d3b09a18062a","Type":"ContainerStarted","Data":"0830b03a12e164dd2a1d967d52b56ead489afd070d43f1aff15ee441cd51303f"} Jul 20 02:58:45.770501 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.770468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bltbd"] Jul 20 02:58:45.772087 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:45.772057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f000f6_0a19_4e6c_aded_34677ef8fc3d.slice/crio-f0a0725525a8c4295651b198605f36c0a4f85dd63af43ea8f9aa1a053ee69c78 WatchSource:0}: Error finding container f0a0725525a8c4295651b198605f36c0a4f85dd63af43ea8f9aa1a053ee69c78: Status 404 returned error can't find the container with id f0a0725525a8c4295651b198605f36c0a4f85dd63af43ea8f9aa1a053ee69c78 Jul 20 02:58:45.807805 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:45.807780 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hk2g6"] Jul 20 02:58:46.161908 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:58:46.161872 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773f6e47_f1d3_4555_8680_c23e0ce263c3.slice/crio-ffb0e9dc8754282c64da538de61b2b836ce696e981043c5bc62a8a1c651a9dea WatchSource:0}: Error finding container ffb0e9dc8754282c64da538de61b2b836ce696e981043c5bc62a8a1c651a9dea: Status 404 returned error can't find the container with id ffb0e9dc8754282c64da538de61b2b836ce696e981043c5bc62a8a1c651a9dea Jul 20 02:58:46.683887 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:46.683847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" event={"ID":"c59ddee8-502d-4ee4-b037-d3b09a18062a","Type":"ContainerStarted","Data":"36af27e9a521ca78f2562b34d2102018681684715a319dd60d00bf2367901b05"} Jul 20 02:58:46.684392 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:46.684354 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:46.686749 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:46.686719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" event={"ID":"773f6e47-f1d3-4555-8680-c23e0ce263c3","Type":"ContainerStarted","Data":"ffb0e9dc8754282c64da538de61b2b836ce696e981043c5bc62a8a1c651a9dea"} Jul 20 02:58:46.690614 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:46.690550 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" event={"ID":"65f000f6-0a19-4e6c-aded-34677ef8fc3d","Type":"ContainerStarted","Data":"f0a0725525a8c4295651b198605f36c0a4f85dd63af43ea8f9aa1a053ee69c78"} Jul 20 02:58:46.703639 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:46.701371 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" podStartSLOduration=1.376986818 podStartE2EDuration="2.701355292s" totalImagesPullingTime="1.324368474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:46.700098958 +0000 UTC m=+821.575900669" watchObservedRunningTime="2026-07-20 02:58:46.701355292 +0000 UTC m=+821.577157006" Jul 20 02:58:47.704794 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.704757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" event={"ID":"65f000f6-0a19-4e6c-aded-34677ef8fc3d","Type":"ContainerStarted","Data":"760c621cb2bc32a0c687e3009269e24420560cc37851af9b5d26e9ecc90d5a1e"} Jul 20 02:58:47.705239 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.704862 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:58:47.706467 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.706441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" event={"ID":"773f6e47-f1d3-4555-8680-c23e0ce263c3","Type":"ContainerStarted","Data":"af2dedc55b3f2cc344106264a27d7f21f719c00a00d8304c4c6f6c6d66b75765"} Jul 20 02:58:47.706597 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.706579 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:47.720672 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.720628 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" podStartSLOduration=0.944270328 podStartE2EDuration="2.720613578s" totalImagesPullingTime="1.77634325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:47.718834748 +0000 UTC m=+822.594636459" watchObservedRunningTime="2026-07-20 02:58:47.720613578 +0000 UTC m=+822.596415287" Jul 20 02:58:47.734433 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:47.734388 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" podStartSLOduration=1.340004803 podStartE2EDuration="2.73437497s" totalImagesPullingTime="1.394370167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:58:47.732285297 +0000 UTC m=+822.608087010" watchObservedRunningTime="2026-07-20 02:58:47.73437497 +0000 UTC m=+822.610176680" Jul 20 02:58:52.712407 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:52.712373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79b94c59c6-fzswd" Jul 20 02:58:58.713711 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:58.713671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-hk2g6" Jul 20 02:58:58.714156 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:58:58.714090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bltbd" Jul 20 02:59:06.031102 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.031058 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489"] Jul 20 02:59:06.038576 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.038553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.041099 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.041070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 20 02:59:06.041391 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.041347 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 02:59:06.041504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.041412 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 02:59:06.041504 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.041421 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6tcpl\"" Jul 20 02:59:06.043933 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.043908 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489"] Jul 20 02:59:06.177461 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.177618 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvm74\" (UniqueName: \"kubernetes.io/projected/6b343014-6632-4b93-b7c3-ea62ae71e40a-kube-api-access-rvm74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.177618 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b343014-6632-4b93-b7c3-ea62ae71e40a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.177618 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.177726 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.177726 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.177641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279036 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279224 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvm74\" (UniqueName: \"kubernetes.io/projected/6b343014-6632-4b93-b7c3-ea62ae71e40a-kube-api-access-rvm74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279503 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b343014-6632-4b93-b7c3-ea62ae71e40a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279503 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279503 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279680 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.279737 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.279695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.281434 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.281387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b343014-6632-4b93-b7c3-ea62ae71e40a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.281827 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.281807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b343014-6632-4b93-b7c3-ea62ae71e40a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.286933 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.286906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvm74\" (UniqueName: \"kubernetes.io/projected/6b343014-6632-4b93-b7c3-ea62ae71e40a-kube-api-access-rvm74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489\" (UID: \"6b343014-6632-4b93-b7c3-ea62ae71e40a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.350100 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.350064 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:06.487692 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.487660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489"] Jul 20 02:59:06.489575 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:59:06.489547 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b343014_6632_4b93_b7c3_ea62ae71e40a.slice/crio-af85c66c30a67599e2e322679e9da13854341a0dd6cfda1e41e1d07f32a3c0f3 WatchSource:0}: Error finding container af85c66c30a67599e2e322679e9da13854341a0dd6cfda1e41e1d07f32a3c0f3: Status 404 returned error can't find the container with id af85c66c30a67599e2e322679e9da13854341a0dd6cfda1e41e1d07f32a3c0f3 Jul 20 02:59:06.812598 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:06.812557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" event={"ID":"6b343014-6632-4b93-b7c3-ea62ae71e40a","Type":"ContainerStarted","Data":"af85c66c30a67599e2e322679e9da13854341a0dd6cfda1e41e1d07f32a3c0f3"} Jul 20 02:59:07.819979 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:07.819949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" event={"ID":"6b343014-6632-4b93-b7c3-ea62ae71e40a","Type":"ContainerStarted","Data":"50ab86594edfc60dc138ebf1a1ae0da8708761423a1035828ebeab3750ef0338"} Jul 20 02:59:07.835488 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:07.835432 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" podStartSLOduration=0.625117381 podStartE2EDuration="1.835415691s" totalImagesPullingTime="1.21029831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:59:07.833243239 +0000 UTC m=+842.709044964" watchObservedRunningTime="2026-07-20 02:59:07.835415691 +0000 UTC m=+842.711217401" Jul 20 02:59:16.351332 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:16.351273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:16.351827 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:16.351389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:16.363888 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:16.363862 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:16.887599 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:16.887571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489" Jul 20 02:59:24.924648 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:24.924592 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh"] Jul 20 02:59:24.928573 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:24.928545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:24.930647 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:24.930614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 20 02:59:24.936090 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:24.936064 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh"] Jul 20 02:59:25.054402 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.054402 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.054656 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a98653c-abbf-4d61-8214-4f78189b440b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.054656 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.054656 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzr6m\" (UniqueName: \"kubernetes.io/projected/7a98653c-abbf-4d61-8214-4f78189b440b-kube-api-access-xzr6m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.054833 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.054681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156052 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a98653c-abbf-4d61-8214-4f78189b440b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzr6m\" (UniqueName: \"kubernetes.io/projected/7a98653c-abbf-4d61-8214-4f78189b440b-kube-api-access-xzr6m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156232 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156641 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156760 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.156846 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.156808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.158855 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.158833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a98653c-abbf-4d61-8214-4f78189b440b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.159263 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.159232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a98653c-abbf-4d61-8214-4f78189b440b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.163128 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.163108 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzr6m\" (UniqueName: \"kubernetes.io/projected/7a98653c-abbf-4d61-8214-4f78189b440b-kube-api-access-xzr6m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2m4nh\" (UID: \"7a98653c-abbf-4d61-8214-4f78189b440b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.242072 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.241982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:25.374386 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.374357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh"] Jul 20 02:59:25.376207 ip-10-0-134-38 kubenswrapper[2565]: W0720 02:59:25.376179 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a98653c_abbf_4d61_8214_4f78189b440b.slice/crio-e29213569cbfe2a97f6a773360d774538b4a732226450ec240c9b8072140ef92 WatchSource:0}: Error finding container e29213569cbfe2a97f6a773360d774538b4a732226450ec240c9b8072140ef92: Status 404 returned error can't find the container with id e29213569cbfe2a97f6a773360d774538b4a732226450ec240c9b8072140ef92 Jul 20 02:59:25.925238 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.925197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" event={"ID":"7a98653c-abbf-4d61-8214-4f78189b440b","Type":"ContainerStarted","Data":"5becb46555d15203f643bcb802c2c8c24a2d59e7f08dd9d2af5f35ccd249b209"} Jul 20 02:59:25.925635 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.925260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" event={"ID":"7a98653c-abbf-4d61-8214-4f78189b440b","Type":"ContainerStarted","Data":"e29213569cbfe2a97f6a773360d774538b4a732226450ec240c9b8072140ef92"} Jul 20 02:59:25.958962 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:25.958901 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" podStartSLOduration=1.809174804 podStartE2EDuration="1.958885884s" totalImagesPullingTime="149.71108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:59:25.955859905 +0000 UTC m=+860.831661613" watchObservedRunningTime="2026-07-20 02:59:25.958885884 +0000 UTC m=+860.834687594" Jul 20 02:59:35.243091 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:35.243051 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:35.243579 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:35.243123 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:35.255804 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:35.255777 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 02:59:35.988864 ip-10-0-134-38 kubenswrapper[2565]: I0720 02:59:35.988830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2m4nh" Jul 20 03:00:00.133963 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.133876 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6"] Jul 20 03:00:00.139650 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.139624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" Jul 20 03:00:00.141848 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.141780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:00:00.142694 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.142671 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6"] Jul 20 03:00:00.174551 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.174526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" Jul 20 03:00:00.320180 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:00.320153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6"] Jul 20 03:00:00.322592 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:00:00.322567 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2975a3da_4598_499b_b380_c917778d31d4.slice/crio-eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30 WatchSource:0}: Error finding container eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30: Status 404 returned error can't find the container with id eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30 Jul 20 03:00:01.112625 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.112590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" event={"ID":"2975a3da-4598-499b-b380-c917778d31d4","Type":"ContainerStarted","Data":"eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30"} Jul 20 03:00:01.420398 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.420319 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f"] Jul 20 03:00:01.423845 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.423821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.425707 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.425682 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 20 03:00:01.431931 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.431902 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f"] Jul 20 03:00:01.587759 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.587922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.587922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.587922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfstx\" (UniqueName: \"kubernetes.io/projected/54886eb1-f2b7-47be-bb68-bbad8d957bc8-kube-api-access-lfstx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.587922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.587922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.587902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689347 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689347 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfstx\" (UniqueName: \"kubernetes.io/projected/54886eb1-f2b7-47be-bb68-bbad8d957bc8-kube-api-access-lfstx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689558 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689558 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689558 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689729 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689785 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.689844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.689809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.691801 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.691770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54886eb1-f2b7-47be-bb68-bbad8d957bc8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.691983 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.691967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54886eb1-f2b7-47be-bb68-bbad8d957bc8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.697187 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.697164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfstx\" (UniqueName: \"kubernetes.io/projected/54886eb1-f2b7-47be-bb68-bbad8d957bc8-kube-api-access-lfstx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cdq4f\" (UID: \"54886eb1-f2b7-47be-bb68-bbad8d957bc8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.736243 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.736209 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:01.894825 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:01.894798 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f"] Jul 20 03:00:01.896676 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:00:01.896635 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54886eb1_f2b7_47be_bb68_bbad8d957bc8.slice/crio-b490df035c88a31d89ca349da82333ca02dd8e2b55419188155264f88019cbec WatchSource:0}: Error finding container b490df035c88a31d89ca349da82333ca02dd8e2b55419188155264f88019cbec: Status 404 returned error can't find the container with id b490df035c88a31d89ca349da82333ca02dd8e2b55419188155264f88019cbec Jul 20 03:00:02.120610 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:02.120574 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" event={"ID":"54886eb1-f2b7-47be-bb68-bbad8d957bc8","Type":"ContainerStarted","Data":"b490df035c88a31d89ca349da82333ca02dd8e2b55419188155264f88019cbec"} Jul 20 03:00:04.133485 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:04.133445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" event={"ID":"54886eb1-f2b7-47be-bb68-bbad8d957bc8","Type":"ContainerStarted","Data":"093a49b27d19c865b267d3af26dfd3f867f139048e5854d8f9ce693ccbe8d361"} Jul 20 03:00:04.148945 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:04.148900 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" podStartSLOduration=1.780313922 podStartE2EDuration="3.148884281s" totalImagesPullingTime="1.368570359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:00:04.147256414 +0000 UTC m=+899.023058127" watchObservedRunningTime="2026-07-20 03:00:04.148884281 +0000 UTC m=+899.024685982" Jul 20 03:00:05.141325 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:05.141274 2565 generic.go:356] "Generic (PLEG): container finished" podID="2975a3da-4598-499b-b380-c917778d31d4" containerID="cc0cc6b6e39360a32f36e981cccbcbd33337341ee2aa68e8fcd5dc986f398499" exitCode=0 Jul 20 03:00:05.141728 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:05.141366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" event={"ID":"2975a3da-4598-499b-b380-c917778d31d4","Type":"ContainerDied","Data":"cc0cc6b6e39360a32f36e981cccbcbd33337341ee2aa68e8fcd5dc986f398499"} Jul 20 03:00:06.290674 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:06.290647 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" Jul 20 03:00:06.990885 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:06.990853 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:00:06.992345 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:06.992282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:00:07.000211 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:07.000186 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:00:07.001410 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:07.001389 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:00:07.154632 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:07.154606 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" Jul 20 03:00:07.154786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:07.154654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741940-x6vg6" event={"ID":"2975a3da-4598-499b-b380-c917778d31d4","Type":"ContainerDied","Data":"eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30"} Jul 20 03:00:07.154786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:07.154726 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eebef4f8a2e4d5b1e72fc2b7c5b3223b4199f3db2b8058a302e5779184050b30" Jul 20 03:00:11.737111 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:11.737075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:11.739535 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:11.737163 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:11.749745 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:11.749719 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:12.196033 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:12.196004 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cdq4f" Jul 20 03:00:14.335531 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.335493 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf"] Jul 20 03:00:14.335910 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.335890 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2975a3da-4598-499b-b380-c917778d31d4" containerName="cleanup" Jul 20 03:00:14.335910 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.335900 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2975a3da-4598-499b-b380-c917778d31d4" containerName="cleanup" Jul 20 03:00:14.335982 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.335967 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2975a3da-4598-499b-b380-c917778d31d4" containerName="cleanup" Jul 20 03:00:14.338909 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.338892 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.340874 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.340851 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 20 03:00:14.347151 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.347129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf"] Jul 20 03:00:14.411976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.411944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.412164 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.411999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.412164 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.412061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.412164 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.412125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.412164 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.412157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.412339 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.412175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64wvt\" (UniqueName: \"kubernetes.io/projected/118eb4e8-4fec-41ce-97e5-d032ec7dc661-kube-api-access-64wvt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513242 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513210 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wvt\" (UniqueName: \"kubernetes.io/projected/118eb4e8-4fec-41ce-97e5-d032ec7dc661-kube-api-access-64wvt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513752 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.513837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.513791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.515569 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.515547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118eb4e8-4fec-41ce-97e5-d032ec7dc661-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.515881 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.515865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118eb4e8-4fec-41ce-97e5-d032ec7dc661-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.520794 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.520774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wvt\" (UniqueName: \"kubernetes.io/projected/118eb4e8-4fec-41ce-97e5-d032ec7dc661-kube-api-access-64wvt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf\" (UID: \"118eb4e8-4fec-41ce-97e5-d032ec7dc661\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.648883 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.648789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:14.778533 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:14.778503 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf"] Jul 20 03:00:14.779374 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:00:14.779351 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118eb4e8_4fec_41ce_97e5_d032ec7dc661.slice/crio-ee21cf7699c38eed5a917451b6f4b7bef7abdd48585731fcf2dc0508d73160d9 WatchSource:0}: Error finding container ee21cf7699c38eed5a917451b6f4b7bef7abdd48585731fcf2dc0508d73160d9: Status 404 returned error can't find the container with id ee21cf7699c38eed5a917451b6f4b7bef7abdd48585731fcf2dc0508d73160d9 Jul 20 03:00:15.203785 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:15.203750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" event={"ID":"118eb4e8-4fec-41ce-97e5-d032ec7dc661","Type":"ContainerStarted","Data":"ee21cf7699c38eed5a917451b6f4b7bef7abdd48585731fcf2dc0508d73160d9"} Jul 20 03:00:23.248422 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:23.248384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" event={"ID":"118eb4e8-4fec-41ce-97e5-d032ec7dc661","Type":"ContainerStarted","Data":"3ddc09903357f375369f62bb1a1f2f54fd358bd4c5b7f48a5d851febcee66049"} Jul 20 03:00:23.264178 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:23.264127 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" podStartSLOduration=1.494396509 podStartE2EDuration="9.264110096s" totalImagesPullingTime="7.769713587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:00:23.262225034 +0000 UTC m=+918.138026747" watchObservedRunningTime="2026-07-20 03:00:23.264110096 +0000 UTC m=+918.139911846" Jul 20 03:00:24.649384 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:24.649348 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:24.649827 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:24.649407 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:24.662035 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:24.662011 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:00:25.272001 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:00:25.271970 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf" Jul 20 03:01:33.316709 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.316628 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544c679c6-2h7xn"] Jul 20 03:01:33.319822 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.319799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.326673 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.326648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544c679c6-2h7xn"] Jul 20 03:01:33.346618 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.346585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2n6p\" (UniqueName: \"kubernetes.io/projected/2eb292be-3f9b-42a0-8faa-3968243d0f03-kube-api-access-b2n6p\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.346751 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.346650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2eb292be-3f9b-42a0-8faa-3968243d0f03-tls-cert\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.447830 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.447787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2n6p\" (UniqueName: \"kubernetes.io/projected/2eb292be-3f9b-42a0-8faa-3968243d0f03-kube-api-access-b2n6p\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.448033 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.447869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2eb292be-3f9b-42a0-8faa-3968243d0f03-tls-cert\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.450367 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.450342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2eb292be-3f9b-42a0-8faa-3968243d0f03-tls-cert\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.454806 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.454784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2n6p\" (UniqueName: \"kubernetes.io/projected/2eb292be-3f9b-42a0-8faa-3968243d0f03-kube-api-access-b2n6p\") pod \"authorino-544c679c6-2h7xn\" (UID: \"2eb292be-3f9b-42a0-8faa-3968243d0f03\") " pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.630826 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.630791 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544c679c6-2h7xn" Jul 20 03:01:33.777776 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.774837 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544c679c6-2h7xn"] Jul 20 03:01:33.778352 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:33.778326 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:01:34.654734 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.654600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544c679c6-2h7xn" event={"ID":"2eb292be-3f9b-42a0-8faa-3968243d0f03","Type":"ContainerStarted","Data":"2078a022e5675374b0214800eca81d1cab27c4db4defbe6d68b5974058f9ba1d"} Jul 20 03:01:34.654734 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.654655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544c679c6-2h7xn" event={"ID":"2eb292be-3f9b-42a0-8faa-3968243d0f03","Type":"ContainerStarted","Data":"77faee3381ce6adcf9514af6a199a808d3dfe0b2ce5d203ca39baab7c3b6fb57"} Jul 20 03:01:34.671907 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.671846 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-544c679c6-2h7xn" podStartSLOduration=1.047303998 podStartE2EDuration="1.671825763s" totalImagesPullingTime="624.521765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:01:34.66864266 +0000 UTC m=+989.544444383" watchObservedRunningTime="2026-07-20 03:01:34.671825763 +0000 UTC m=+989.547627474" Jul 20 03:01:34.693065 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.693027 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 03:01:34.693261 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.693201 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" podUID="22e16fca-d5f5-4714-9221-95561d750aaa" containerName="authorino" containerID="cri-o://65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb" gracePeriod=30 Jul 20 03:01:34.962203 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:34.962174 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 03:01:35.058573 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.058541 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf84x\" (UniqueName: \"kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x\") pod \"22e16fca-d5f5-4714-9221-95561d750aaa\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " Jul 20 03:01:35.058755 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.058589 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert\") pod \"22e16fca-d5f5-4714-9221-95561d750aaa\" (UID: \"22e16fca-d5f5-4714-9221-95561d750aaa\") " Jul 20 03:01:35.061223 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.061193 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x" (OuterVolumeSpecName: "kube-api-access-vf84x") pod "22e16fca-d5f5-4714-9221-95561d750aaa" (UID: "22e16fca-d5f5-4714-9221-95561d750aaa"). InnerVolumeSpecName "kube-api-access-vf84x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:01:35.069676 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.069650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "22e16fca-d5f5-4714-9221-95561d750aaa" (UID: "22e16fca-d5f5-4714-9221-95561d750aaa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:01:35.166316 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.160240 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf84x\" (UniqueName: \"kubernetes.io/projected/22e16fca-d5f5-4714-9221-95561d750aaa-kube-api-access-vf84x\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:01:35.166316 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.160276 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e16fca-d5f5-4714-9221-95561d750aaa-tls-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:01:35.661692 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.661656 2565 generic.go:356] "Generic (PLEG): container finished" podID="22e16fca-d5f5-4714-9221-95561d750aaa" containerID="65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb" exitCode=0 Jul 20 03:01:35.662166 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.661720 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" Jul 20 03:01:35.662166 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.661749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" event={"ID":"22e16fca-d5f5-4714-9221-95561d750aaa","Type":"ContainerDied","Data":"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb"} Jul 20 03:01:35.662166 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.661808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5655c9f75c-bbq8k" event={"ID":"22e16fca-d5f5-4714-9221-95561d750aaa","Type":"ContainerDied","Data":"8b00c747e97d80fad0fef53521f22e984771dcad1b1b8ac0a24fa8f56536b765"} Jul 20 03:01:35.662166 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.661831 2565 scope.go:117] "RemoveContainer" containerID="65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb" Jul 20 03:01:35.672190 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.672169 2565 scope.go:117] "RemoveContainer" containerID="65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb" Jul 20 03:01:35.672496 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:01:35.672473 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb\": container with ID starting with 65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb not found: ID does not exist" containerID="65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb" Jul 20 03:01:35.672602 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.672514 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb"} err="failed to get container status \"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb\": rpc error: code = NotFound desc = could not find container \"65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb\": container with ID starting with 65abbc859c10170abf7857fbbee3ca7a2e48f5f3514748a36e9ab831e344addb not found: ID does not exist" Jul 20 03:01:35.680016 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.679983 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 03:01:35.683713 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:35.683689 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5655c9f75c-bbq8k"] Jul 20 03:01:37.193548 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:01:37.193514 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e16fca-d5f5-4714-9221-95561d750aaa" path="/var/lib/kubelet/pods/22e16fca-d5f5-4714-9221-95561d750aaa/volumes" Jul 20 03:02:20.593012 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.592972 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 03:02:20.593523 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.593185 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" podUID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" containerName="manager" containerID="cri-o://65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18" gracePeriod=30 Jul 20 03:02:20.840928 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.840901 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 03:02:20.887566 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.887474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") pod \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " Jul 20 03:02:20.887731 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.887575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shwrt\" (UniqueName: \"kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt\") pod \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\" (UID: \"5e54b88a-49ab-4888-a5ac-6c6e97f04aea\") " Jul 20 03:02:20.889595 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.889559 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5e54b88a-49ab-4888-a5ac-6c6e97f04aea" (UID: "5e54b88a-49ab-4888-a5ac-6c6e97f04aea"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:02:20.889726 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.889698 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt" (OuterVolumeSpecName: "kube-api-access-shwrt") pod "5e54b88a-49ab-4888-a5ac-6c6e97f04aea" (UID: "5e54b88a-49ab-4888-a5ac-6c6e97f04aea"). InnerVolumeSpecName "kube-api-access-shwrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:02:20.925960 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.925926 2565 generic.go:356] "Generic (PLEG): container finished" podID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" containerID="65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18" exitCode=0 Jul 20 03:02:20.926110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.925965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" event={"ID":"5e54b88a-49ab-4888-a5ac-6c6e97f04aea","Type":"ContainerDied","Data":"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18"} Jul 20 03:02:20.926110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.926020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" event={"ID":"5e54b88a-49ab-4888-a5ac-6c6e97f04aea","Type":"ContainerDied","Data":"a0cafb9f80b2da889d6775bad8a868ae9e9347cd97174cc1b3cfb73d304a1d58"} Jul 20 03:02:20.926110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.926043 2565 scope.go:117] "RemoveContainer" containerID="65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18" Jul 20 03:02:20.926110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.925973 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-k4tgk" Jul 20 03:02:20.935578 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.935560 2565 scope.go:117] "RemoveContainer" containerID="65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18" Jul 20 03:02:20.935809 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:02:20.935788 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18\": container with ID starting with 65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18 not found: ID does not exist" containerID="65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18" Jul 20 03:02:20.935856 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.935821 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18"} err="failed to get container status \"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18\": rpc error: code = NotFound desc = could not find container \"65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18\": container with ID starting with 65992a70334027b2172a655f883a27a709815937f06bfeb6c59df9401aba4b18 not found: ID does not exist" Jul 20 03:02:20.946185 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.946156 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 03:02:20.949941 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.949919 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-k4tgk"] Jul 20 03:02:20.988574 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.988542 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-webhook-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:02:20.988574 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:20.988568 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwrt\" (UniqueName: \"kubernetes.io/projected/5e54b88a-49ab-4888-a5ac-6c6e97f04aea-kube-api-access-shwrt\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:02:21.192256 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:02:21.192174 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" path="/var/lib/kubelet/pods/5e54b88a-49ab-4888-a5ac-6c6e97f04aea/volumes" Jul 20 03:03:21.745081 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745045 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:21.745606 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745553 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e16fca-d5f5-4714-9221-95561d750aaa" containerName="authorino" Jul 20 03:03:21.745606 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745580 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e16fca-d5f5-4714-9221-95561d750aaa" containerName="authorino" Jul 20 03:03:21.745786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745616 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" containerName="manager" Jul 20 03:03:21.745786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745627 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" containerName="manager" Jul 20 03:03:21.745786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745740 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e16fca-d5f5-4714-9221-95561d750aaa" containerName="authorino" Jul 20 03:03:21.745786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.745755 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e54b88a-49ab-4888-a5ac-6c6e97f04aea" containerName="manager" Jul 20 03:03:21.748905 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.748888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.751444 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.751426 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 03:03:21.751543 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.751425 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 03:03:21.751543 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.751426 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb24c\"" Jul 20 03:03:21.758616 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.758594 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:21.857128 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.857091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.857344 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.857167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghljt\" (UniqueName: \"kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.958512 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.958477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljt\" (UniqueName: \"kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.958662 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.958569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.960937 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.960919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:21.965975 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:21.965943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghljt\" (UniqueName: \"kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt\") pod \"maas-controller-645d8c8bc6-mfvtv\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:22.060461 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:22.060379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:22.198549 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:22.198523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:22.201485 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:03:22.201460 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b93f52d_8525_4df1_b2e9_f8f878c8c384.slice/crio-4a4382411d69fce07153e76ffd47eebbbc81a67757b67e1aa06ad53ec09a79b2 WatchSource:0}: Error finding container 4a4382411d69fce07153e76ffd47eebbbc81a67757b67e1aa06ad53ec09a79b2: Status 404 returned error can't find the container with id 4a4382411d69fce07153e76ffd47eebbbc81a67757b67e1aa06ad53ec09a79b2 Jul 20 03:03:22.262899 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:22.262863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" event={"ID":"9b93f52d-8525-4df1-b2e9-f8f878c8c384","Type":"ContainerStarted","Data":"4a4382411d69fce07153e76ffd47eebbbc81a67757b67e1aa06ad53ec09a79b2"} Jul 20 03:03:23.270350 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:23.270314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" event={"ID":"9b93f52d-8525-4df1-b2e9-f8f878c8c384","Type":"ContainerStarted","Data":"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c"} Jul 20 03:03:23.270846 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:23.270426 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:23.285768 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:23.285714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" podStartSLOduration=1.789164408 podStartE2EDuration="2.285696282s" totalImagesPullingTime="496.531874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:03:23.284458291 +0000 UTC m=+1098.160260002" watchObservedRunningTime="2026-07-20 03:03:23.285696282 +0000 UTC m=+1098.161497994" Jul 20 03:03:34.278493 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:34.278457 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:38.909638 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:38.909601 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:38.910054 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:38.909820 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" podUID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" containerName="manager" containerID="cri-o://802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c" gracePeriod=30 Jul 20 03:03:39.161363 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.161286 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:39.338319 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.338263 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghljt\" (UniqueName: \"kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt\") pod \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " Jul 20 03:03:39.338522 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.338360 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert\") pod \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\" (UID: \"9b93f52d-8525-4df1-b2e9-f8f878c8c384\") " Jul 20 03:03:39.340473 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.340434 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt" (OuterVolumeSpecName: "kube-api-access-ghljt") pod "9b93f52d-8525-4df1-b2e9-f8f878c8c384" (UID: "9b93f52d-8525-4df1-b2e9-f8f878c8c384"). InnerVolumeSpecName "kube-api-access-ghljt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:03:39.340590 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.340468 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9b93f52d-8525-4df1-b2e9-f8f878c8c384" (UID: "9b93f52d-8525-4df1-b2e9-f8f878c8c384"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:03:39.359833 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.359802 2565 generic.go:356] "Generic (PLEG): container finished" podID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" containerID="802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c" exitCode=0 Jul 20 03:03:39.359962 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.359854 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" Jul 20 03:03:39.359962 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.359881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" event={"ID":"9b93f52d-8525-4df1-b2e9-f8f878c8c384","Type":"ContainerDied","Data":"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c"} Jul 20 03:03:39.359962 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.359926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-mfvtv" event={"ID":"9b93f52d-8525-4df1-b2e9-f8f878c8c384","Type":"ContainerDied","Data":"4a4382411d69fce07153e76ffd47eebbbc81a67757b67e1aa06ad53ec09a79b2"} Jul 20 03:03:39.359962 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.359949 2565 scope.go:117] "RemoveContainer" containerID="802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c" Jul 20 03:03:39.369377 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.369346 2565 scope.go:117] "RemoveContainer" containerID="802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c" Jul 20 03:03:39.369687 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:03:39.369660 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c\": container with ID starting with 802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c not found: ID does not exist" containerID="802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c" Jul 20 03:03:39.369772 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.369692 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c"} err="failed to get container status \"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c\": rpc error: code = NotFound desc = could not find container \"802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c\": container with ID starting with 802f025edc47c2107e98eed86b2914bad72846e4695a7698c7acad39834fd75c not found: ID does not exist" Jul 20 03:03:39.381539 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.381517 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:39.383798 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.383778 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-mfvtv"] Jul 20 03:03:39.439562 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.439476 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghljt\" (UniqueName: \"kubernetes.io/projected/9b93f52d-8525-4df1-b2e9-f8f878c8c384-kube-api-access-ghljt\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:03:39.439562 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:39.439508 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b93f52d-8525-4df1-b2e9-f8f878c8c384-webhook-cert\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:03:40.222799 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.222760 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-hxh5g"] Jul 20 03:03:40.223252 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.223157 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" containerName="manager" Jul 20 03:03:40.223252 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.223173 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" containerName="manager" Jul 20 03:03:40.223252 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.223244 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" containerName="manager" Jul 20 03:03:40.227532 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.227514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.233130 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.233108 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 03:03:40.233335 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.233109 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lb24c\"" Jul 20 03:03:40.233335 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.233147 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 03:03:40.247258 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.247230 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-hxh5g"] Jul 20 03:03:40.348236 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.348199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fda65796-3ea2-47ec-8b5f-b6db2b173d31-webhook-cert\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.348439 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.348243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mjx\" (UniqueName: \"kubernetes.io/projected/fda65796-3ea2-47ec-8b5f-b6db2b173d31-kube-api-access-66mjx\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.449455 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.449422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mjx\" (UniqueName: \"kubernetes.io/projected/fda65796-3ea2-47ec-8b5f-b6db2b173d31-kube-api-access-66mjx\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.449625 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.449535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fda65796-3ea2-47ec-8b5f-b6db2b173d31-webhook-cert\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.452031 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.452006 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fda65796-3ea2-47ec-8b5f-b6db2b173d31-webhook-cert\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.457353 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.457321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mjx\" (UniqueName: \"kubernetes.io/projected/fda65796-3ea2-47ec-8b5f-b6db2b173d31-kube-api-access-66mjx\") pod \"maas-controller-645d8c8bc6-hxh5g\" (UID: \"fda65796-3ea2-47ec-8b5f-b6db2b173d31\") " pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.537816 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.537737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:40.674769 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:40.674743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d8c8bc6-hxh5g"] Jul 20 03:03:40.676777 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:03:40.676752 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda65796_3ea2_47ec_8b5f_b6db2b173d31.slice/crio-ea49232e98b6e4bb3955288b869f72ffda52caf82d28e0dbebf96d3377c6fe56 WatchSource:0}: Error finding container ea49232e98b6e4bb3955288b869f72ffda52caf82d28e0dbebf96d3377c6fe56: Status 404 returned error can't find the container with id ea49232e98b6e4bb3955288b869f72ffda52caf82d28e0dbebf96d3377c6fe56 Jul 20 03:03:41.195459 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:41.193968 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b93f52d-8525-4df1-b2e9-f8f878c8c384" path="/var/lib/kubelet/pods/9b93f52d-8525-4df1-b2e9-f8f878c8c384/volumes" Jul 20 03:03:41.374138 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:41.374095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" event={"ID":"fda65796-3ea2-47ec-8b5f-b6db2b173d31","Type":"ContainerStarted","Data":"cb8bee1ccee8d39994ff84d0f7f8b21409330f91c898ffb39220f36fce508fbf"} Jul 20 03:03:41.374569 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:41.374160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" event={"ID":"fda65796-3ea2-47ec-8b5f-b6db2b173d31","Type":"ContainerStarted","Data":"ea49232e98b6e4bb3955288b869f72ffda52caf82d28e0dbebf96d3377c6fe56"} Jul 20 03:03:41.374569 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:41.374212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:03:41.390991 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:41.390943 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" podStartSLOduration=1.086803167 podStartE2EDuration="1.390926993s" totalImagesPullingTime="304.123826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:03:41.388369973 +0000 UTC m=+1116.264171685" watchObservedRunningTime="2026-07-20 03:03:41.390926993 +0000 UTC m=+1116.266728725" Jul 20 03:03:52.382697 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:03:52.382667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d8c8bc6-hxh5g" Jul 20 03:05:07.035664 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:05:07.035636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:05:07.038543 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:05:07.038518 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:05:07.042638 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:05:07.042615 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:05:07.045350 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:05:07.045331 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:08:25.592729 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:25.592688 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 03:08:25.593449 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:25.592929 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" podUID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" containerName="manager" containerID="cri-o://e100807d478ee9eb7032d1b0ff9490fd7a314974dc093790394552199d9e2a71" gracePeriod=10 Jul 20 03:08:29.030576 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.030540 2565 generic.go:356] "Generic (PLEG): container finished" podID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" containerID="e100807d478ee9eb7032d1b0ff9490fd7a314974dc093790394552199d9e2a71" exitCode=0 Jul 20 03:08:29.030914 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.030610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" event={"ID":"1572dbee-2256-4071-9bf4-cec53ffb2ccd","Type":"ContainerDied","Data":"e100807d478ee9eb7032d1b0ff9490fd7a314974dc093790394552199d9e2a71"} Jul 20 03:08:29.049168 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.049142 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 03:08:29.096656 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.096624 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume\") pod \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " Jul 20 03:08:29.096856 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.096731 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d7s9\" (UniqueName: \"kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9\") pod \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\" (UID: \"1572dbee-2256-4071-9bf4-cec53ffb2ccd\") " Jul 20 03:08:29.096992 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.096969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1572dbee-2256-4071-9bf4-cec53ffb2ccd" (UID: "1572dbee-2256-4071-9bf4-cec53ffb2ccd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:08:29.097051 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.097041 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1572dbee-2256-4071-9bf4-cec53ffb2ccd-extensions-socket-volume\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:08:29.098818 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.098796 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9" (OuterVolumeSpecName: "kube-api-access-6d7s9") pod "1572dbee-2256-4071-9bf4-cec53ffb2ccd" (UID: "1572dbee-2256-4071-9bf4-cec53ffb2ccd"). InnerVolumeSpecName "kube-api-access-6d7s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:08:29.198089 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:29.197829 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7s9\" (UniqueName: \"kubernetes.io/projected/1572dbee-2256-4071-9bf4-cec53ffb2ccd-kube-api-access-6d7s9\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:08:30.037306 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:30.037265 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" Jul 20 03:08:30.037791 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:30.037264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk" event={"ID":"1572dbee-2256-4071-9bf4-cec53ffb2ccd","Type":"ContainerDied","Data":"e80a347ce8aa40cc207e7c131ddd1b70fec51b717befbf68e533961058d7405b"} Jul 20 03:08:30.037791 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:30.037441 2565 scope.go:117] "RemoveContainer" containerID="e100807d478ee9eb7032d1b0ff9490fd7a314974dc093790394552199d9e2a71" Jul 20 03:08:30.054076 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:30.054050 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 03:08:30.057346 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:30.057321 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hsgk"] Jul 20 03:08:31.192546 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:08:31.192508 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" path="/var/lib/kubelet/pods/1572dbee-2256-4071-9bf4-cec53ffb2ccd/volumes" Jul 20 03:09:26.802183 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.802145 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr"] Jul 20 03:09:26.802666 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.802580 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" containerName="manager" Jul 20 03:09:26.802666 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.802594 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" containerName="manager" Jul 20 03:09:26.802666 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.802650 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1572dbee-2256-4071-9bf4-cec53ffb2ccd" containerName="manager" Jul 20 03:09:26.806952 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.806934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:26.809074 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.809050 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c9sxh\"" Jul 20 03:09:26.817404 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.817382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr"] Jul 20 03:09:26.924274 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.924232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsctq\" (UniqueName: \"kubernetes.io/projected/f8419e8d-8365-4e0f-aae0-e4441d115b95-kube-api-access-fsctq\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:26.924488 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:26.924333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8419e8d-8365-4e0f-aae0-e4441d115b95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.025610 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.025566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsctq\" (UniqueName: \"kubernetes.io/projected/f8419e8d-8365-4e0f-aae0-e4441d115b95-kube-api-access-fsctq\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.025781 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.025638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8419e8d-8365-4e0f-aae0-e4441d115b95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.026053 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.026029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8419e8d-8365-4e0f-aae0-e4441d115b95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.033107 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.033082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsctq\" (UniqueName: \"kubernetes.io/projected/f8419e8d-8365-4e0f-aae0-e4441d115b95-kube-api-access-fsctq\") pod \"kuadrant-operator-controller-manager-86878c769f-vqwhr\" (UID: \"f8419e8d-8365-4e0f-aae0-e4441d115b95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.118381 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.118334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.265787 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.265756 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr"] Jul 20 03:09:27.267504 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:09:27.267475 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8419e8d_8365_4e0f_aae0_e4441d115b95.slice/crio-a6b0850f1a14a63a4ec5802acfd7dc1fec6d6e36b3bc3f0927daf2df386b665c WatchSource:0}: Error finding container a6b0850f1a14a63a4ec5802acfd7dc1fec6d6e36b3bc3f0927daf2df386b665c: Status 404 returned error can't find the container with id a6b0850f1a14a63a4ec5802acfd7dc1fec6d6e36b3bc3f0927daf2df386b665c Jul 20 03:09:27.347992 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.347960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" event={"ID":"f8419e8d-8365-4e0f-aae0-e4441d115b95","Type":"ContainerStarted","Data":"38183cc6ebf7c86e1747347e7658ddb25afb912bf567b876a195138b16f7a42a"} Jul 20 03:09:27.348105 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.348020 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:09:27.348105 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.348039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" event={"ID":"f8419e8d-8365-4e0f-aae0-e4441d115b95","Type":"ContainerStarted","Data":"a6b0850f1a14a63a4ec5802acfd7dc1fec6d6e36b3bc3f0927daf2df386b665c"} Jul 20 03:09:27.368163 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:27.368113 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" podStartSLOduration=1.368095415 podStartE2EDuration="1.368095415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:09:27.366604028 +0000 UTC m=+1462.242405741" watchObservedRunningTime="2026-07-20 03:09:27.368095415 +0000 UTC m=+1462.243897125" Jul 20 03:09:38.356430 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:09:38.356396 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqwhr" Jul 20 03:10:07.070734 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:10:07.070698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:10:07.074912 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:10:07.074884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:10:07.077714 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:10:07.077692 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:10:07.081705 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:10:07.081687 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:15:07.110803 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:15:07.110776 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:15:07.115874 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:15:07.115840 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:15:07.116793 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:15:07.116763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:15:07.122817 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:15:07.122796 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:17:09.075392 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.075356 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:09.079256 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.079230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.081263 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.081237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-33118b8e-serving-cert\"" Jul 20 03:17:09.088050 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.087731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:09.153475 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.153441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.153663 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.153507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xzj\" (UniqueName: \"kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.153663 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.153574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.254994 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.254958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.255208 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.255043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.255208 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.255136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xzj\" (UniqueName: \"kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.255208 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:17:09.255192 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-serving-cert: secret "maas-api-e2e-ait-33118b8e-serving-cert" not found Jul 20 03:17:09.255445 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:17:09.255280 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls podName:e710c198-8831-452a-9cfd-e2168f56cbdf nodeName:}" failed. No retries permitted until 2026-07-20 03:17:09.755256144 +0000 UTC m=+1924.631057847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls") pod "maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" (UID: "e710c198-8831-452a-9cfd-e2168f56cbdf") : secret "maas-api-e2e-ait-33118b8e-serving-cert" not found Jul 20 03:17:09.255727 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.255708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.263497 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.263467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xzj\" (UniqueName: \"kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.760679 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.760642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.763175 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.763157 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") pod \"maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:09.991358 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:09.991317 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:10.338914 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.338883 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:10.340534 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:17:10.340504 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode710c198_8831_452a_9cfd_e2168f56cbdf.slice/crio-3fbb011f18c4652a887eeafdf009e8aabacf2d039ccb7f54712b117ee4468b6f WatchSource:0}: Error finding container 3fbb011f18c4652a887eeafdf009e8aabacf2d039ccb7f54712b117ee4468b6f: Status 404 returned error can't find the container with id 3fbb011f18c4652a887eeafdf009e8aabacf2d039ccb7f54712b117ee4468b6f Jul 20 03:17:10.342040 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.342017 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:17:10.925348 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.925314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" event={"ID":"e710c198-8831-452a-9cfd-e2168f56cbdf","Type":"ContainerStarted","Data":"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908"} Jul 20 03:17:10.925509 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.925372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" event={"ID":"e710c198-8831-452a-9cfd-e2168f56cbdf","Type":"ContainerStarted","Data":"3fbb011f18c4652a887eeafdf009e8aabacf2d039ccb7f54712b117ee4468b6f"} Jul 20 03:17:10.925509 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.925393 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:10.941093 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:10.941048 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" podStartSLOduration=1.448890811 podStartE2EDuration="1.941032151s" totalImagesPullingTime="492.14134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:17:10.939452272 +0000 UTC m=+1925.815253994" watchObservedRunningTime="2026-07-20 03:17:10.941032151 +0000 UTC m=+1925.816833883" Jul 20 03:17:16.936934 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:16.936853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:45.131453 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.131417 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:45.134029 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.132166 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" podUID="e710c198-8831-452a-9cfd-e2168f56cbdf" containerName="maas-api" containerID="cri-o://65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908" gracePeriod=30 Jul 20 03:17:45.391736 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.391675 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:45.514180 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.514147 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4xzj\" (UniqueName: \"kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj\") pod \"e710c198-8831-452a-9cfd-e2168f56cbdf\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " Jul 20 03:17:45.514413 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.514199 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") pod \"e710c198-8831-452a-9cfd-e2168f56cbdf\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " Jul 20 03:17:45.514413 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.514273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca\") pod \"e710c198-8831-452a-9cfd-e2168f56cbdf\" (UID: \"e710c198-8831-452a-9cfd-e2168f56cbdf\") " Jul 20 03:17:45.514642 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.514609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e710c198-8831-452a-9cfd-e2168f56cbdf" (UID: "e710c198-8831-452a-9cfd-e2168f56cbdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:17:45.516469 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.516447 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e710c198-8831-452a-9cfd-e2168f56cbdf" (UID: "e710c198-8831-452a-9cfd-e2168f56cbdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:17:45.516553 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.516535 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj" (OuterVolumeSpecName: "kube-api-access-l4xzj") pod "e710c198-8831-452a-9cfd-e2168f56cbdf" (UID: "e710c198-8831-452a-9cfd-e2168f56cbdf"). InnerVolumeSpecName "kube-api-access-l4xzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:17:45.615468 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.615435 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4xzj\" (UniqueName: \"kubernetes.io/projected/e710c198-8831-452a-9cfd-e2168f56cbdf-kube-api-access-l4xzj\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:17:45.615468 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.615466 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e710c198-8831-452a-9cfd-e2168f56cbdf-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:17:45.615468 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:45.615475 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e710c198-8831-452a-9cfd-e2168f56cbdf-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:17:46.119195 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.119162 2565 generic.go:356] "Generic (PLEG): container finished" podID="e710c198-8831-452a-9cfd-e2168f56cbdf" containerID="65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908" exitCode=0 Jul 20 03:17:46.119416 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.119212 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" Jul 20 03:17:46.119416 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.119212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" event={"ID":"e710c198-8831-452a-9cfd-e2168f56cbdf","Type":"ContainerDied","Data":"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908"} Jul 20 03:17:46.119416 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.119363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc" event={"ID":"e710c198-8831-452a-9cfd-e2168f56cbdf","Type":"ContainerDied","Data":"3fbb011f18c4652a887eeafdf009e8aabacf2d039ccb7f54712b117ee4468b6f"} Jul 20 03:17:46.119416 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.119400 2565 scope.go:117] "RemoveContainer" containerID="65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908" Jul 20 03:17:46.129768 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.129747 2565 scope.go:117] "RemoveContainer" containerID="65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908" Jul 20 03:17:46.130056 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:17:46.130021 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908\": container with ID starting with 65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908 not found: ID does not exist" containerID="65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908" Jul 20 03:17:46.130134 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.130062 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908"} err="failed to get container status \"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908\": rpc error: code = NotFound desc = could not find container \"65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908\": container with ID starting with 65fc61e32060e5aec191cfe155bab90aabaac31918f3e6dfaa7d2e3d7cb92908 not found: ID does not exist" Jul 20 03:17:46.140739 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.140710 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:46.143060 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:46.143036 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33118b8e-59bfbb87cd-tmgsc"] Jul 20 03:17:47.191796 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:47.191759 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e710c198-8831-452a-9cfd-e2168f56cbdf" path="/var/lib/kubelet/pods/e710c198-8831-452a-9cfd-e2168f56cbdf/volumes" Jul 20 03:17:53.323484 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.323437 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:17:53.324422 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.324399 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e710c198-8831-452a-9cfd-e2168f56cbdf" containerName="maas-api" Jul 20 03:17:53.324422 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.324422 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e710c198-8831-452a-9cfd-e2168f56cbdf" containerName="maas-api" Jul 20 03:17:53.324649 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.324631 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e710c198-8831-452a-9cfd-e2168f56cbdf" containerName="maas-api" Jul 20 03:17:53.330387 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.330322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.333860 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.333839 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-53dce522-serving-cert\"" Jul 20 03:17:53.342086 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.342061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:17:53.493830 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.493795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.494030 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.493901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.494030 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.493927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpjk\" (UniqueName: \"kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.595180 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.595147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.595403 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.595234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.595403 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.595252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpjk\" (UniqueName: \"kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.595546 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:17:53.595395 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-serving-cert: secret "maas-api-e2e-migrate-53dce522-serving-cert" not found Jul 20 03:17:53.595546 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:17:53.595475 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls podName:c2e009db-d5c5-4228-afa5-61fdaf6eaca3 nodeName:}" failed. No retries permitted until 2026-07-20 03:17:54.095452481 +0000 UTC m=+1968.971254173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls") pod "maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" (UID: "c2e009db-d5c5-4228-afa5-61fdaf6eaca3") : secret "maas-api-e2e-migrate-53dce522-serving-cert" not found Jul 20 03:17:53.595844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.595823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:53.603171 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:53.603147 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpjk\" (UniqueName: \"kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:54.101411 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:54.101376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:54.103771 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:54.103747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") pod \"maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:54.241238 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:54.241205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:54.380602 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:54.380564 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:17:54.381473 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:17:54.381446 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e009db_d5c5_4228_afa5_61fdaf6eaca3.slice/crio-9b1cbc8751278419353766fa6fb9b2c57dd75556bcf3971f670101bc4f196adb WatchSource:0}: Error finding container 9b1cbc8751278419353766fa6fb9b2c57dd75556bcf3971f670101bc4f196adb: Status 404 returned error can't find the container with id 9b1cbc8751278419353766fa6fb9b2c57dd75556bcf3971f670101bc4f196adb Jul 20 03:17:55.179193 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:55.179149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" event={"ID":"c2e009db-d5c5-4228-afa5-61fdaf6eaca3","Type":"ContainerStarted","Data":"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a"} Jul 20 03:17:55.179395 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:55.179204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" event={"ID":"c2e009db-d5c5-4228-afa5-61fdaf6eaca3","Type":"ContainerStarted","Data":"9b1cbc8751278419353766fa6fb9b2c57dd75556bcf3971f670101bc4f196adb"} Jul 20 03:17:55.179395 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:55.179226 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:17:55.196366 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:17:55.196319 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" podStartSLOduration=1.802695707 podStartE2EDuration="2.19630372s" totalImagesPullingTime="393.608013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:17:55.193992689 +0000 UTC m=+1970.069794400" watchObservedRunningTime="2026-07-20 03:17:55.19630372 +0000 UTC m=+1970.072105421" Jul 20 03:18:01.192229 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:01.192158 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:18:28.615262 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.615222 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:18:28.615858 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.615508 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" podUID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" containerName="maas-api" containerID="cri-o://988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a" gracePeriod=30 Jul 20 03:18:28.866325 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.866243 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:18:28.916667 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.916634 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") pod \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " Jul 20 03:18:28.916844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.916725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skpjk\" (UniqueName: \"kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk\") pod \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " Jul 20 03:18:28.916844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.916778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca\") pod \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\" (UID: \"c2e009db-d5c5-4228-afa5-61fdaf6eaca3\") " Jul 20 03:18:28.917138 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.917114 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2e009db-d5c5-4228-afa5-61fdaf6eaca3" (UID: "c2e009db-d5c5-4228-afa5-61fdaf6eaca3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:18:28.918684 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.918658 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2e009db-d5c5-4228-afa5-61fdaf6eaca3" (UID: "c2e009db-d5c5-4228-afa5-61fdaf6eaca3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:18:28.918684 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:28.918674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk" (OuterVolumeSpecName: "kube-api-access-skpjk") pod "c2e009db-d5c5-4228-afa5-61fdaf6eaca3" (UID: "c2e009db-d5c5-4228-afa5-61fdaf6eaca3"). InnerVolumeSpecName "kube-api-access-skpjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:18:29.017945 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.017912 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:18:29.017945 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.017940 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skpjk\" (UniqueName: \"kubernetes.io/projected/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-kube-api-access-skpjk\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:18:29.017945 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.017951 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e009db-d5c5-4228-afa5-61fdaf6eaca3-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:18:29.377267 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.377220 2565 generic.go:356] "Generic (PLEG): container finished" podID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" containerID="988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a" exitCode=0 Jul 20 03:18:29.377441 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.377344 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" Jul 20 03:18:29.377441 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.377350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" event={"ID":"c2e009db-d5c5-4228-afa5-61fdaf6eaca3","Type":"ContainerDied","Data":"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a"} Jul 20 03:18:29.377535 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.377470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr" event={"ID":"c2e009db-d5c5-4228-afa5-61fdaf6eaca3","Type":"ContainerDied","Data":"9b1cbc8751278419353766fa6fb9b2c57dd75556bcf3971f670101bc4f196adb"} Jul 20 03:18:29.377535 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.377502 2565 scope.go:117] "RemoveContainer" containerID="988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a" Jul 20 03:18:29.388828 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.388804 2565 scope.go:117] "RemoveContainer" containerID="988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a" Jul 20 03:18:29.389345 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:18:29.389316 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a\": container with ID starting with 988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a not found: ID does not exist" containerID="988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a" Jul 20 03:18:29.389444 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.389354 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a"} err="failed to get container status \"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a\": rpc error: code = NotFound desc = could not find container \"988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a\": container with ID starting with 988e6b52babd4a255f75cadbfe78dd77c6054847006e265afbe6883006c9183a not found: ID does not exist" Jul 20 03:18:29.392876 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.392855 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:18:29.397888 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:29.397862 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-53dce522-7c9f89f86f-wv4zr"] Jul 20 03:18:31.192512 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:31.192477 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" path="/var/lib/kubelet/pods/c2e009db-d5c5-4228-afa5-61fdaf6eaca3/volumes" Jul 20 03:18:36.525856 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.525811 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:18:36.526269 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.526254 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" containerName="maas-api" Jul 20 03:18:36.526358 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.526272 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" containerName="maas-api" Jul 20 03:18:36.526406 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.526395 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e009db-d5c5-4228-afa5-61fdaf6eaca3" containerName="maas-api" Jul 20 03:18:36.529437 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.529418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.531347 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.531321 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dd7b8c7f-serving-cert\"" Jul 20 03:18:36.537200 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.537169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:18:36.692774 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.692728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.692934 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.692885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.692985 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.692953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9fn\" (UniqueName: \"kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.794333 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.794226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9fn\" (UniqueName: \"kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.794333 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.794320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.794528 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.794434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.794528 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:18:36.794461 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-serving-cert: secret "maas-api-e2e-ait-dd7b8c7f-serving-cert" not found Jul 20 03:18:36.794605 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:18:36.794555 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls podName:9f35eb8f-93dc-497d-b706-cff12c123e77 nodeName:}" failed. No retries permitted until 2026-07-20 03:18:37.294537324 +0000 UTC m=+2012.170339016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls") pod "maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" (UID: "9f35eb8f-93dc-497d-b706-cff12c123e77") : secret "maas-api-e2e-ait-dd7b8c7f-serving-cert" not found Jul 20 03:18:36.795064 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.795046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:36.804407 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:36.804382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9fn\" (UniqueName: \"kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:37.299844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:37.299804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:37.302370 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:37.302350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") pod \"maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:37.443440 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:37.443406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:37.787790 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:37.787761 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:18:37.789372 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:18:37.789335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f35eb8f_93dc_497d_b706_cff12c123e77.slice/crio-fbe740abd98fdc525b78ed4efef3aed66f7f05d9cf0dc4cb6dbdb58b82408138 WatchSource:0}: Error finding container fbe740abd98fdc525b78ed4efef3aed66f7f05d9cf0dc4cb6dbdb58b82408138: Status 404 returned error can't find the container with id fbe740abd98fdc525b78ed4efef3aed66f7f05d9cf0dc4cb6dbdb58b82408138 Jul 20 03:18:38.429689 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:38.429648 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" event={"ID":"9f35eb8f-93dc-497d-b706-cff12c123e77","Type":"ContainerStarted","Data":"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412"} Jul 20 03:18:38.429879 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:38.429718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" event={"ID":"9f35eb8f-93dc-497d-b706-cff12c123e77","Type":"ContainerStarted","Data":"fbe740abd98fdc525b78ed4efef3aed66f7f05d9cf0dc4cb6dbdb58b82408138"} Jul 20 03:18:38.429879 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:38.429860 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:18:38.444564 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:38.444519 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" podStartSLOduration=1.980368178 podStartE2EDuration="2.444502589s" totalImagesPullingTime="464.134411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:18:38.443461884 +0000 UTC m=+2013.319263608" watchObservedRunningTime="2026-07-20 03:18:38.444502589 +0000 UTC m=+2013.320304300" Jul 20 03:18:44.440545 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:18:44.440517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:19:12.620505 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.620469 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:19:12.621048 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.620724 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" podUID="9f35eb8f-93dc-497d-b706-cff12c123e77" containerName="maas-api" containerID="cri-o://3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412" gracePeriod=30 Jul 20 03:19:12.871994 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.871930 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:19:12.917260 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.917227 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") pod \"9f35eb8f-93dc-497d-b706-cff12c123e77\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " Jul 20 03:19:12.917410 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.917281 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca\") pod \"9f35eb8f-93dc-497d-b706-cff12c123e77\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " Jul 20 03:19:12.917474 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.917462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx9fn\" (UniqueName: \"kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn\") pod \"9f35eb8f-93dc-497d-b706-cff12c123e77\" (UID: \"9f35eb8f-93dc-497d-b706-cff12c123e77\") " Jul 20 03:19:12.917674 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.917650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f35eb8f-93dc-497d-b706-cff12c123e77" (UID: "9f35eb8f-93dc-497d-b706-cff12c123e77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:19:12.917789 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.917770 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f35eb8f-93dc-497d-b706-cff12c123e77-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:12.919212 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.919193 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f35eb8f-93dc-497d-b706-cff12c123e77" (UID: "9f35eb8f-93dc-497d-b706-cff12c123e77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:19:12.919472 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:12.919443 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn" (OuterVolumeSpecName: "kube-api-access-cx9fn") pod "9f35eb8f-93dc-497d-b706-cff12c123e77" (UID: "9f35eb8f-93dc-497d-b706-cff12c123e77"). InnerVolumeSpecName "kube-api-access-cx9fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:19:13.019122 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.019089 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx9fn\" (UniqueName: \"kubernetes.io/projected/9f35eb8f-93dc-497d-b706-cff12c123e77-kube-api-access-cx9fn\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:13.019122 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.019118 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f35eb8f-93dc-497d-b706-cff12c123e77-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:13.629433 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.629396 2565 generic.go:356] "Generic (PLEG): container finished" podID="9f35eb8f-93dc-497d-b706-cff12c123e77" containerID="3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412" exitCode=0 Jul 20 03:19:13.629842 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.629457 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" Jul 20 03:19:13.629842 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.629484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" event={"ID":"9f35eb8f-93dc-497d-b706-cff12c123e77","Type":"ContainerDied","Data":"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412"} Jul 20 03:19:13.629842 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.629539 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg" event={"ID":"9f35eb8f-93dc-497d-b706-cff12c123e77","Type":"ContainerDied","Data":"fbe740abd98fdc525b78ed4efef3aed66f7f05d9cf0dc4cb6dbdb58b82408138"} Jul 20 03:19:13.629842 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.629561 2565 scope.go:117] "RemoveContainer" containerID="3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412" Jul 20 03:19:13.638475 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.638455 2565 scope.go:117] "RemoveContainer" containerID="3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412" Jul 20 03:19:13.638739 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:19:13.638719 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412\": container with ID starting with 3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412 not found: ID does not exist" containerID="3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412" Jul 20 03:19:13.638800 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.638746 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412"} err="failed to get container status \"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412\": rpc error: code = NotFound desc = could not find container \"3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412\": container with ID starting with 3541e93b9c409a04f04aed54b7a2c0b47c678cc210ab428599ae98aabcff0412 not found: ID does not exist" Jul 20 03:19:13.643683 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.643662 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:19:13.646895 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:13.646874 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd7b8c7f-644699f8bb-2grwg"] Jul 20 03:19:15.192484 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:15.192450 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f35eb8f-93dc-497d-b706-cff12c123e77" path="/var/lib/kubelet/pods/9f35eb8f-93dc-497d-b706-cff12c123e77/volumes" Jul 20 03:19:21.641171 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.641137 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:21.641646 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.641629 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f35eb8f-93dc-497d-b706-cff12c123e77" containerName="maas-api" Jul 20 03:19:21.641693 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.641649 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f35eb8f-93dc-497d-b706-cff12c123e77" containerName="maas-api" Jul 20 03:19:21.641734 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.641724 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f35eb8f-93dc-497d-b706-cff12c123e77" containerName="maas-api" Jul 20 03:19:21.645719 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.645696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.647798 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.647774 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-45fa7d8a-serving-cert\"" Jul 20 03:19:21.656931 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.656810 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:21.703011 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.702973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.703178 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.703051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6bk\" (UniqueName: \"kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.703178 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.703119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.804511 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.804471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6bk\" (UniqueName: \"kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.804705 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.804532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.804705 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.804610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.804705 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:19:21.804633 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-serving-cert: secret "maas-api-e2e-derive-45fa7d8a-serving-cert" not found Jul 20 03:19:21.804705 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:19:21.804689 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls podName:62592a5a-c8ed-4d76-a701-377885a44ee2 nodeName:}" failed. No retries permitted until 2026-07-20 03:19:22.304672603 +0000 UTC m=+2057.180474295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls") pod "maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" (UID: "62592a5a-c8ed-4d76-a701-377885a44ee2") : secret "maas-api-e2e-derive-45fa7d8a-serving-cert" not found Jul 20 03:19:21.805264 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.805244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:21.813084 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:21.813058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6bk\" (UniqueName: \"kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:22.309620 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:22.309588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:22.311984 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:22.311957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") pod \"maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:22.557835 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:22.557800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:22.699384 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:22.699353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:22.701014 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:19:22.700984 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62592a5a_c8ed_4d76_a701_377885a44ee2.slice/crio-99423190cbf0888b8da2e2a8b0d5ec15ff2f0cb89fa9a0e28ce0d0a708617ce4 WatchSource:0}: Error finding container 99423190cbf0888b8da2e2a8b0d5ec15ff2f0cb89fa9a0e28ce0d0a708617ce4: Status 404 returned error can't find the container with id 99423190cbf0888b8da2e2a8b0d5ec15ff2f0cb89fa9a0e28ce0d0a708617ce4 Jul 20 03:19:23.688876 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:23.688843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" event={"ID":"62592a5a-c8ed-4d76-a701-377885a44ee2","Type":"ContainerStarted","Data":"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844"} Jul 20 03:19:23.689068 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:23.688892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" event={"ID":"62592a5a-c8ed-4d76-a701-377885a44ee2","Type":"ContainerStarted","Data":"99423190cbf0888b8da2e2a8b0d5ec15ff2f0cb89fa9a0e28ce0d0a708617ce4"} Jul 20 03:19:23.689068 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:23.688973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:23.706542 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:23.706492 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" podStartSLOduration=2.340780379 podStartE2EDuration="2.706477397s" totalImagesPullingTime="365.697018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:19:23.702652719 +0000 UTC m=+2058.578454419" watchObservedRunningTime="2026-07-20 03:19:23.706477397 +0000 UTC m=+2058.582279106" Jul 20 03:19:29.700954 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:29.700882 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:57.017873 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.017834 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:57.018922 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.018883 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" podUID="62592a5a-c8ed-4d76-a701-377885a44ee2" containerName="maas-api" containerID="cri-o://77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844" gracePeriod=30 Jul 20 03:19:57.272688 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.272632 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:57.331636 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.331610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") pod \"62592a5a-c8ed-4d76-a701-377885a44ee2\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " Jul 20 03:19:57.331788 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.331769 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca\") pod \"62592a5a-c8ed-4d76-a701-377885a44ee2\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " Jul 20 03:19:57.331829 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.331812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql6bk\" (UniqueName: \"kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk\") pod \"62592a5a-c8ed-4d76-a701-377885a44ee2\" (UID: \"62592a5a-c8ed-4d76-a701-377885a44ee2\") " Jul 20 03:19:57.332110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.332082 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62592a5a-c8ed-4d76-a701-377885a44ee2" (UID: "62592a5a-c8ed-4d76-a701-377885a44ee2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:19:57.333766 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.333739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62592a5a-c8ed-4d76-a701-377885a44ee2" (UID: "62592a5a-c8ed-4d76-a701-377885a44ee2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:19:57.333849 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.333791 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk" (OuterVolumeSpecName: "kube-api-access-ql6bk") pod "62592a5a-c8ed-4d76-a701-377885a44ee2" (UID: "62592a5a-c8ed-4d76-a701-377885a44ee2"). InnerVolumeSpecName "kube-api-access-ql6bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:19:57.432662 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.432623 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6bk\" (UniqueName: \"kubernetes.io/projected/62592a5a-c8ed-4d76-a701-377885a44ee2-kube-api-access-ql6bk\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:57.432662 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.432657 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62592a5a-c8ed-4d76-a701-377885a44ee2-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:57.432662 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.432667 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62592a5a-c8ed-4d76-a701-377885a44ee2-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:19:57.892158 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.892121 2565 generic.go:356] "Generic (PLEG): container finished" podID="62592a5a-c8ed-4d76-a701-377885a44ee2" containerID="77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844" exitCode=0 Jul 20 03:19:57.892158 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.892183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" Jul 20 03:19:57.892466 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.892196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" event={"ID":"62592a5a-c8ed-4d76-a701-377885a44ee2","Type":"ContainerDied","Data":"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844"} Jul 20 03:19:57.892466 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.892346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl" event={"ID":"62592a5a-c8ed-4d76-a701-377885a44ee2","Type":"ContainerDied","Data":"99423190cbf0888b8da2e2a8b0d5ec15ff2f0cb89fa9a0e28ce0d0a708617ce4"} Jul 20 03:19:57.892466 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.892378 2565 scope.go:117] "RemoveContainer" containerID="77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844" Jul 20 03:19:57.902472 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.902453 2565 scope.go:117] "RemoveContainer" containerID="77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844" Jul 20 03:19:57.902748 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:19:57.902727 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844\": container with ID starting with 77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844 not found: ID does not exist" containerID="77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844" Jul 20 03:19:57.902797 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.902757 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844"} err="failed to get container status \"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844\": rpc error: code = NotFound desc = could not find container \"77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844\": container with ID starting with 77f8cb3839f93bbd657754810bc4e4165e25f8de5076070bda2803ec78a75844 not found: ID does not exist" Jul 20 03:19:57.913113 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.913090 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:57.915414 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:57.915394 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45fa7d8a-78cc899f4-vvlbl"] Jul 20 03:19:59.192004 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:19:59.191967 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62592a5a-c8ed-4d76-a701-377885a44ee2" path="/var/lib/kubelet/pods/62592a5a-c8ed-4d76-a701-377885a44ee2/volumes" Jul 20 03:20:07.145708 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:20:07.145681 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:20:07.151862 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:20:07.151840 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:20:07.152404 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:20:07.152381 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:20:07.157741 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:20:07.157725 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:21:48.026844 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.026795 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:21:48.027480 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.027429 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62592a5a-c8ed-4d76-a701-377885a44ee2" containerName="maas-api" Jul 20 03:21:48.027480 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.027450 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62592a5a-c8ed-4d76-a701-377885a44ee2" containerName="maas-api" Jul 20 03:21:48.027626 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.027582 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62592a5a-c8ed-4d76-a701-377885a44ee2" containerName="maas-api" Jul 20 03:21:48.030996 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.030970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.033765 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.033149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1cc4bbf4-serving-cert\"" Jul 20 03:21:48.046913 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.046884 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:21:48.180369 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.180332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.180555 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.180377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.180555 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.180416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvdh\" (UniqueName: \"kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.282026 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.281929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.282026 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.281978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.282026 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.282017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvdh\" (UniqueName: \"kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.282740 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.282717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.284837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.284812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.289870 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.289848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvdh\" (UniqueName: \"kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh\") pod \"maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.345184 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.345147 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:48.505011 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:48.504928 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:21:48.506995 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:21:48.506955 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a2ef9a_a149_4eef_bd56_fb02317b17b0.slice/crio-8cf2b7ada2857a510a2c6ae862c9561fe92f434a4fcbb6ff65aaef9e9b71a7c6 WatchSource:0}: Error finding container 8cf2b7ada2857a510a2c6ae862c9561fe92f434a4fcbb6ff65aaef9e9b71a7c6: Status 404 returned error can't find the container with id 8cf2b7ada2857a510a2c6ae862c9561fe92f434a4fcbb6ff65aaef9e9b71a7c6 Jul 20 03:21:49.519805 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:49.519768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" event={"ID":"c1a2ef9a-a149-4eef-bd56-fb02317b17b0","Type":"ContainerStarted","Data":"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664"} Jul 20 03:21:49.520417 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:49.519824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" event={"ID":"c1a2ef9a-a149-4eef-bd56-fb02317b17b0","Type":"ContainerStarted","Data":"8cf2b7ada2857a510a2c6ae862c9561fe92f434a4fcbb6ff65aaef9e9b71a7c6"} Jul 20 03:21:49.520417 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:49.519865 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:21:49.537854 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:49.537810 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" podStartSLOduration=1.017583867 podStartE2EDuration="1.537795737s" totalImagesPullingTime="520.21187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:21:49.535852898 +0000 UTC m=+2204.411654608" watchObservedRunningTime="2026-07-20 03:21:49.537795737 +0000 UTC m=+2204.413597447" Jul 20 03:21:55.532698 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:21:55.532669 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:22:10.076156 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.076118 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:22:10.076589 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.076440 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" podUID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" containerName="maas-api" containerID="cri-o://053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664" gracePeriod=30 Jul 20 03:22:10.327794 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.327737 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:22:10.374881 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.374840 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqvdh\" (UniqueName: \"kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh\") pod \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " Jul 20 03:22:10.375042 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.374914 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls\") pod \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " Jul 20 03:22:10.375042 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.374953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca\") pod \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\" (UID: \"c1a2ef9a-a149-4eef-bd56-fb02317b17b0\") " Jul 20 03:22:10.375580 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.375548 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1a2ef9a-a149-4eef-bd56-fb02317b17b0" (UID: "c1a2ef9a-a149-4eef-bd56-fb02317b17b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:22:10.377026 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.376995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh" (OuterVolumeSpecName: "kube-api-access-mqvdh") pod "c1a2ef9a-a149-4eef-bd56-fb02317b17b0" (UID: "c1a2ef9a-a149-4eef-bd56-fb02317b17b0"). InnerVolumeSpecName "kube-api-access-mqvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:22:10.377471 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.377448 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1a2ef9a-a149-4eef-bd56-fb02317b17b0" (UID: "c1a2ef9a-a149-4eef-bd56-fb02317b17b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:22:10.476434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.476397 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:22:10.476434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.476425 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqvdh\" (UniqueName: \"kubernetes.io/projected/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-kube-api-access-mqvdh\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:22:10.476434 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.476435 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a2ef9a-a149-4eef-bd56-fb02317b17b0-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:22:10.647046 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.646953 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" containerID="053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664" exitCode=0 Jul 20 03:22:10.647046 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.646997 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" Jul 20 03:22:10.647046 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.647017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" event={"ID":"c1a2ef9a-a149-4eef-bd56-fb02317b17b0","Type":"ContainerDied","Data":"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664"} Jul 20 03:22:10.647350 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.647059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs" event={"ID":"c1a2ef9a-a149-4eef-bd56-fb02317b17b0","Type":"ContainerDied","Data":"8cf2b7ada2857a510a2c6ae862c9561fe92f434a4fcbb6ff65aaef9e9b71a7c6"} Jul 20 03:22:10.647350 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.647080 2565 scope.go:117] "RemoveContainer" containerID="053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664" Jul 20 03:22:10.656932 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.656909 2565 scope.go:117] "RemoveContainer" containerID="053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664" Jul 20 03:22:10.657154 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:22:10.657131 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664\": container with ID starting with 053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664 not found: ID does not exist" containerID="053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664" Jul 20 03:22:10.657213 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.657163 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664"} err="failed to get container status \"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664\": rpc error: code = NotFound desc = could not find container \"053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664\": container with ID starting with 053f2e6eb53bc6162fe3df3a32985388c2ba5b6f2cbba4df827343127ed61664 not found: ID does not exist" Jul 20 03:22:10.666821 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.666792 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:22:10.671326 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:10.671275 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1cc4bbf4-f6944bc47-vc5cs"] Jul 20 03:22:11.192414 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:11.192377 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" path="/var/lib/kubelet/pods/c1a2ef9a-a149-4eef-bd56-fb02317b17b0/volumes" Jul 20 03:22:43.251320 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.251275 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:22:43.251900 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.251792 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" containerName="maas-api" Jul 20 03:22:43.251900 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.251812 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" containerName="maas-api" Jul 20 03:22:43.252064 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.251902 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a2ef9a-a149-4eef-bd56-fb02317b17b0" containerName="maas-api" Jul 20 03:22:43.254766 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.254748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.256929 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.256905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27c7795f-serving-cert\"" Jul 20 03:22:43.263835 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.263811 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:22:43.391723 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.391687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.391896 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.391753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.391896 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.391825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9pd\" (UniqueName: \"kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.493182 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.493137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.493421 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.493242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9pd\" (UniqueName: \"kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.493421 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:22:43.493321 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-serving-cert: secret "maas-api-e2e-mt-27c7795f-serving-cert" not found Jul 20 03:22:43.493421 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.493335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.493421 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:22:43.493412 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls podName:4fd409c5-40f6-4af2-90c2-e7221571fae0 nodeName:}" failed. No retries permitted until 2026-07-20 03:22:43.993396237 +0000 UTC m=+2258.869197931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls") pod "maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" (UID: "4fd409c5-40f6-4af2-90c2-e7221571fae0") : secret "maas-api-e2e-mt-27c7795f-serving-cert" not found Jul 20 03:22:43.494009 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.493986 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.501573 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.501514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9pd\" (UniqueName: \"kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:43.998162 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:43.998127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:44.000781 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.000748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") pod \"maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:44.168785 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.168746 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:44.314948 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.314922 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:22:44.317149 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:22:44.317122 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd409c5_40f6_4af2_90c2_e7221571fae0.slice/crio-e2be50b18ec2ce834ce96246e7735e350c8d8d92834450283b6cf587efb90438 WatchSource:0}: Error finding container e2be50b18ec2ce834ce96246e7735e350c8d8d92834450283b6cf587efb90438: Status 404 returned error can't find the container with id e2be50b18ec2ce834ce96246e7735e350c8d8d92834450283b6cf587efb90438 Jul 20 03:22:44.318614 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.318595 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:22:44.842866 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.842833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" event={"ID":"4fd409c5-40f6-4af2-90c2-e7221571fae0","Type":"ContainerStarted","Data":"9546149bd90935298ce1fb85ce5787f4846111bd00a7d877523118660affa269"} Jul 20 03:22:44.843041 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.842887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" event={"ID":"4fd409c5-40f6-4af2-90c2-e7221571fae0","Type":"ContainerStarted","Data":"e2be50b18ec2ce834ce96246e7735e350c8d8d92834450283b6cf587efb90438"} Jul 20 03:22:44.843041 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.842918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:22:44.859884 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:44.859815 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" podStartSLOduration=1.470562481 podStartE2EDuration="1.859798414s" totalImagesPullingTime="389.235933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:22:44.85764982 +0000 UTC m=+2259.733451532" watchObservedRunningTime="2026-07-20 03:22:44.859798414 +0000 UTC m=+2259.735600124" Jul 20 03:22:50.853651 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:22:50.853618 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:23:47.059645 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.059599 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:23:47.060218 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.059858 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" podUID="4fd409c5-40f6-4af2-90c2-e7221571fae0" containerName="maas-api" containerID="cri-o://9546149bd90935298ce1fb85ce5787f4846111bd00a7d877523118660affa269" gracePeriod=30 Jul 20 03:23:47.199187 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.199151 2565 generic.go:356] "Generic (PLEG): container finished" podID="4fd409c5-40f6-4af2-90c2-e7221571fae0" containerID="9546149bd90935298ce1fb85ce5787f4846111bd00a7d877523118660affa269" exitCode=0 Jul 20 03:23:47.199958 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.199933 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" event={"ID":"4fd409c5-40f6-4af2-90c2-e7221571fae0","Type":"ContainerDied","Data":"9546149bd90935298ce1fb85ce5787f4846111bd00a7d877523118660affa269"} Jul 20 03:23:47.310247 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.310183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:23:47.400378 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.400345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") pod \"4fd409c5-40f6-4af2-90c2-e7221571fae0\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " Jul 20 03:23:47.400564 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.400428 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca\") pod \"4fd409c5-40f6-4af2-90c2-e7221571fae0\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " Jul 20 03:23:47.400564 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.400450 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts9pd\" (UniqueName: \"kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd\") pod \"4fd409c5-40f6-4af2-90c2-e7221571fae0\" (UID: \"4fd409c5-40f6-4af2-90c2-e7221571fae0\") " Jul 20 03:23:47.400749 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.400723 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fd409c5-40f6-4af2-90c2-e7221571fae0" (UID: "4fd409c5-40f6-4af2-90c2-e7221571fae0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:23:47.402464 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.402441 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd" (OuterVolumeSpecName: "kube-api-access-ts9pd") pod "4fd409c5-40f6-4af2-90c2-e7221571fae0" (UID: "4fd409c5-40f6-4af2-90c2-e7221571fae0"). InnerVolumeSpecName "kube-api-access-ts9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:23:47.402537 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.402478 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fd409c5-40f6-4af2-90c2-e7221571fae0" (UID: "4fd409c5-40f6-4af2-90c2-e7221571fae0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:23:47.501112 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.501076 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd409c5-40f6-4af2-90c2-e7221571fae0-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:23:47.501112 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.501108 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd409c5-40f6-4af2-90c2-e7221571fae0-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:23:47.501112 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:47.501119 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9pd\" (UniqueName: \"kubernetes.io/projected/4fd409c5-40f6-4af2-90c2-e7221571fae0-kube-api-access-ts9pd\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:23:48.206158 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:48.206127 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" Jul 20 03:23:48.206636 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:48.206127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp" event={"ID":"4fd409c5-40f6-4af2-90c2-e7221571fae0","Type":"ContainerDied","Data":"e2be50b18ec2ce834ce96246e7735e350c8d8d92834450283b6cf587efb90438"} Jul 20 03:23:48.206636 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:48.206259 2565 scope.go:117] "RemoveContainer" containerID="9546149bd90935298ce1fb85ce5787f4846111bd00a7d877523118660affa269" Jul 20 03:23:48.227068 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:48.227043 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:23:48.230748 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:48.230725 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27c7795f-6fc7cfdc48-nqhkp"] Jul 20 03:23:49.193342 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:23:49.193307 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd409c5-40f6-4af2-90c2-e7221571fae0" path="/var/lib/kubelet/pods/4fd409c5-40f6-4af2-90c2-e7221571fae0/volumes" Jul 20 03:25:07.195590 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:07.195555 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:25:07.200479 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:07.200442 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:25:07.203403 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:07.203376 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:25:07.209912 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:07.209890 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:25:47.246134 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.246098 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:25:47.246661 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.246499 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd409c5-40f6-4af2-90c2-e7221571fae0" containerName="maas-api" Jul 20 03:25:47.246661 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.246514 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd409c5-40f6-4af2-90c2-e7221571fae0" containerName="maas-api" Jul 20 03:25:47.246661 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.246592 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd409c5-40f6-4af2-90c2-e7221571fae0" containerName="maas-api" Jul 20 03:25:47.249525 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.249507 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.252510 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.252487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-da51e5-serving-cert\"" Jul 20 03:25:47.270512 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.270485 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:25:47.367783 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.367747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwfsl\" (UniqueName: \"kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.367964 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.367816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.367964 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.367857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.468883 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.468846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.469088 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.468894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.469088 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.468953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwfsl\" (UniqueName: \"kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.469088 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:25:47.469009 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-serving-cert: secret "maas-api-e2e-inf-a-da51e5-serving-cert" not found Jul 20 03:25:47.469409 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:25:47.469106 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls podName:40b1054b-908d-4799-ae7a-55187b576f75 nodeName:}" failed. No retries permitted until 2026-07-20 03:25:47.969082924 +0000 UTC m=+2442.844884615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls") pod "maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" (UID: "40b1054b-908d-4799-ae7a-55187b576f75") : secret "maas-api-e2e-inf-a-da51e5-serving-cert" not found Jul 20 03:25:47.469656 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.469634 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.480478 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.480448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwfsl\" (UniqueName: \"kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.975326 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.975275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:47.977919 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:47.977881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") pod \"maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:48.159419 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.159389 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:48.302826 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.302799 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:25:48.305251 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:25:48.305224 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b1054b_908d_4799_ae7a_55187b576f75.slice/crio-56a0946f510df518e58ded6561b6e2e414b0c9c54db2d39a68f6dbe80be75c5a WatchSource:0}: Error finding container 56a0946f510df518e58ded6561b6e2e414b0c9c54db2d39a68f6dbe80be75c5a: Status 404 returned error can't find the container with id 56a0946f510df518e58ded6561b6e2e414b0c9c54db2d39a68f6dbe80be75c5a Jul 20 03:25:48.881929 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.881888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" event={"ID":"40b1054b-908d-4799-ae7a-55187b576f75","Type":"ContainerStarted","Data":"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463"} Jul 20 03:25:48.882073 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.881955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" event={"ID":"40b1054b-908d-4799-ae7a-55187b576f75","Type":"ContainerStarted","Data":"56a0946f510df518e58ded6561b6e2e414b0c9c54db2d39a68f6dbe80be75c5a"} Jul 20 03:25:48.882073 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.882022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:48.902365 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:48.902314 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" podStartSLOduration=1.440413104 podStartE2EDuration="1.902274745s" totalImagesPullingTime="461.861641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:25:48.900136651 +0000 UTC m=+2443.775938362" watchObservedRunningTime="2026-07-20 03:25:48.902274745 +0000 UTC m=+2443.778076454" Jul 20 03:25:54.896363 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:54.896331 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:25:55.438689 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.438653 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:25:55.442433 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.442410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.444560 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.444534 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9fe24f-serving-cert\"" Jul 20 03:25:55.452805 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.452777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:25:55.541899 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.541861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.542068 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.541907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s6w\" (UniqueName: \"kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.542068 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.541979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.642890 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.642854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.643065 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.643006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.643065 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.643052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s6w\" (UniqueName: \"kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.643656 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.643637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.645600 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.645575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.651802 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.651776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s6w\" (UniqueName: \"kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w\") pod \"maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.756572 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.756493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:55.904330 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.904285 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:25:55.905706 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:25:55.905677 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f40969_092d_4d85_95a3_c8fe263a8187.slice/crio-9994ed4b864b99a1e642d8bd3930a50951769d83c2b3d13df856380ac9609c79 WatchSource:0}: Error finding container 9994ed4b864b99a1e642d8bd3930a50951769d83c2b3d13df856380ac9609c79: Status 404 returned error can't find the container with id 9994ed4b864b99a1e642d8bd3930a50951769d83c2b3d13df856380ac9609c79 Jul 20 03:25:55.925109 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:55.925079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" event={"ID":"f7f40969-092d-4d85-95a3-c8fe263a8187","Type":"ContainerStarted","Data":"9994ed4b864b99a1e642d8bd3930a50951769d83c2b3d13df856380ac9609c79"} Jul 20 03:25:56.932343 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:56.932266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" event={"ID":"f7f40969-092d-4d85-95a3-c8fe263a8187","Type":"ContainerStarted","Data":"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583"} Jul 20 03:25:56.932775 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:56.932493 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:25:56.947444 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:25:56.947398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" podStartSLOduration=1.443910996 podStartE2EDuration="1.947382479s" totalImagesPullingTime="503.471483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:25:56.946435046 +0000 UTC m=+2451.822236756" watchObservedRunningTime="2026-07-20 03:25:56.947382479 +0000 UTC m=+2451.823184190" Jul 20 03:26:02.943337 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:26:02.943285 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:27:20.646402 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:20.646286 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:27:20.646864 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:20.646591 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" podUID="40b1054b-908d-4799-ae7a-55187b576f75" containerName="maas-api" containerID="cri-o://3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463" gracePeriod=30 Jul 20 03:27:20.899712 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:20.899659 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:27:21.055664 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.055629 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwfsl\" (UniqueName: \"kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl\") pod \"40b1054b-908d-4799-ae7a-55187b576f75\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " Jul 20 03:27:21.055863 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.055678 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") pod \"40b1054b-908d-4799-ae7a-55187b576f75\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " Jul 20 03:27:21.055863 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.055699 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca\") pod \"40b1054b-908d-4799-ae7a-55187b576f75\" (UID: \"40b1054b-908d-4799-ae7a-55187b576f75\") " Jul 20 03:27:21.056095 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.056070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40b1054b-908d-4799-ae7a-55187b576f75" (UID: "40b1054b-908d-4799-ae7a-55187b576f75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:27:21.057835 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.057802 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40b1054b-908d-4799-ae7a-55187b576f75" (UID: "40b1054b-908d-4799-ae7a-55187b576f75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:27:21.057835 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.057809 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl" (OuterVolumeSpecName: "kube-api-access-mwfsl") pod "40b1054b-908d-4799-ae7a-55187b576f75" (UID: "40b1054b-908d-4799-ae7a-55187b576f75"). InnerVolumeSpecName "kube-api-access-mwfsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:21.157257 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.157170 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwfsl\" (UniqueName: \"kubernetes.io/projected/40b1054b-908d-4799-ae7a-55187b576f75-kube-api-access-mwfsl\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:21.157257 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.157203 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b1054b-908d-4799-ae7a-55187b576f75-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:21.157257 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.157212 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b1054b-908d-4799-ae7a-55187b576f75-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:21.404890 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.404854 2565 generic.go:356] "Generic (PLEG): container finished" podID="40b1054b-908d-4799-ae7a-55187b576f75" containerID="3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463" exitCode=0 Jul 20 03:27:21.405051 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.404901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" event={"ID":"40b1054b-908d-4799-ae7a-55187b576f75","Type":"ContainerDied","Data":"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463"} Jul 20 03:27:21.405051 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.404906 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" Jul 20 03:27:21.405051 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.404941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67" event={"ID":"40b1054b-908d-4799-ae7a-55187b576f75","Type":"ContainerDied","Data":"56a0946f510df518e58ded6561b6e2e414b0c9c54db2d39a68f6dbe80be75c5a"} Jul 20 03:27:21.405051 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.404963 2565 scope.go:117] "RemoveContainer" containerID="3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463" Jul 20 03:27:21.414153 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.414132 2565 scope.go:117] "RemoveContainer" containerID="3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463" Jul 20 03:27:21.414476 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:27:21.414450 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463\": container with ID starting with 3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463 not found: ID does not exist" containerID="3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463" Jul 20 03:27:21.414586 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.414480 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463"} err="failed to get container status \"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463\": rpc error: code = NotFound desc = could not find container \"3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463\": container with ID starting with 3943243b73f4d2e2ca7f48fbba521856e2bf88f0e688e368f66d2eff19197463 not found: ID does not exist" Jul 20 03:27:21.420521 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.420497 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:27:21.425486 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:21.425463 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da51e5-5bd46b6db5-tzt67"] Jul 20 03:27:23.191806 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:23.191771 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b1054b-908d-4799-ae7a-55187b576f75" path="/var/lib/kubelet/pods/40b1054b-908d-4799-ae7a-55187b576f75/volumes" Jul 20 03:27:42.607923 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.607885 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:27:42.608556 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.608143 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" podUID="f7f40969-092d-4d85-95a3-c8fe263a8187" containerName="maas-api" containerID="cri-o://e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583" gracePeriod=30 Jul 20 03:27:42.861982 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.861919 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:27:42.957698 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.957665 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49s6w\" (UniqueName: \"kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w\") pod \"f7f40969-092d-4d85-95a3-c8fe263a8187\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " Jul 20 03:27:42.957862 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.957748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls\") pod \"f7f40969-092d-4d85-95a3-c8fe263a8187\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " Jul 20 03:27:42.957862 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.957818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca\") pod \"f7f40969-092d-4d85-95a3-c8fe263a8187\" (UID: \"f7f40969-092d-4d85-95a3-c8fe263a8187\") " Jul 20 03:27:42.958136 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.958098 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7f40969-092d-4d85-95a3-c8fe263a8187" (UID: "f7f40969-092d-4d85-95a3-c8fe263a8187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:27:42.959760 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.959730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w" (OuterVolumeSpecName: "kube-api-access-49s6w") pod "f7f40969-092d-4d85-95a3-c8fe263a8187" (UID: "f7f40969-092d-4d85-95a3-c8fe263a8187"). InnerVolumeSpecName "kube-api-access-49s6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:42.959874 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:42.959792 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7f40969-092d-4d85-95a3-c8fe263a8187" (UID: "f7f40969-092d-4d85-95a3-c8fe263a8187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:27:43.058953 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.058923 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f40969-092d-4d85-95a3-c8fe263a8187-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:43.058953 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.058949 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f40969-092d-4d85-95a3-c8fe263a8187-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:43.059135 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.058961 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49s6w\" (UniqueName: \"kubernetes.io/projected/f7f40969-092d-4d85-95a3-c8fe263a8187-kube-api-access-49s6w\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:27:43.528775 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.528673 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7f40969-092d-4d85-95a3-c8fe263a8187" containerID="e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583" exitCode=0 Jul 20 03:27:43.528775 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.528729 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" Jul 20 03:27:43.528775 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.528763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" event={"ID":"f7f40969-092d-4d85-95a3-c8fe263a8187","Type":"ContainerDied","Data":"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583"} Jul 20 03:27:43.529017 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.528817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6" event={"ID":"f7f40969-092d-4d85-95a3-c8fe263a8187","Type":"ContainerDied","Data":"9994ed4b864b99a1e642d8bd3930a50951769d83c2b3d13df856380ac9609c79"} Jul 20 03:27:43.529017 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.528839 2565 scope.go:117] "RemoveContainer" containerID="e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583" Jul 20 03:27:43.538107 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.538086 2565 scope.go:117] "RemoveContainer" containerID="e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583" Jul 20 03:27:43.538362 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:27:43.538343 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583\": container with ID starting with e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583 not found: ID does not exist" containerID="e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583" Jul 20 03:27:43.538441 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.538370 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583"} err="failed to get container status \"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583\": rpc error: code = NotFound desc = could not find container \"e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583\": container with ID starting with e77c39ff13af2ce13dc0acab90e50e06d17ac20c5b6eede64a2fda5b653d9583 not found: ID does not exist" Jul 20 03:27:43.544845 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.544810 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:27:43.548703 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:43.548676 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9fe24f-55b5dc998c-9lbp6"] Jul 20 03:27:45.191408 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:45.191370 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f40969-092d-4d85-95a3-c8fe263a8187" path="/var/lib/kubelet/pods/f7f40969-092d-4d85-95a3-c8fe263a8187/volumes" Jul 20 03:27:57.841246 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841185 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:27:57.841841 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841818 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b1054b-908d-4799-ae7a-55187b576f75" containerName="maas-api" Jul 20 03:27:57.841895 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841845 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b1054b-908d-4799-ae7a-55187b576f75" containerName="maas-api" Jul 20 03:27:57.841895 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841868 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f40969-092d-4d85-95a3-c8fe263a8187" containerName="maas-api" Jul 20 03:27:57.841895 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841873 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f40969-092d-4d85-95a3-c8fe263a8187" containerName="maas-api" Jul 20 03:27:57.841991 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841935 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f40969-092d-4d85-95a3-c8fe263a8187" containerName="maas-api" Jul 20 03:27:57.841991 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.841947 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b1054b-908d-4799-ae7a-55187b576f75" containerName="maas-api" Jul 20 03:27:57.845135 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.845117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:57.847143 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.847116 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-52abd4-serving-cert\"" Jul 20 03:27:57.863784 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.863755 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:27:57.997240 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.997195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:57.997500 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.997258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbb6\" (UniqueName: \"kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:57.997500 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:57.997357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.098541 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.098443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.098541 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.098498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbb6\" (UniqueName: \"kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.098541 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.098526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.098801 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:27:58.098652 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-serving-cert: secret "maas-api-e2e-api-a-52abd4-serving-cert" not found Jul 20 03:27:58.098801 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:27:58.098738 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls podName:1dfe2a94-f678-4802-a954-6d454dab0f5e nodeName:}" failed. No retries permitted until 2026-07-20 03:27:58.598715316 +0000 UTC m=+2573.474517004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls") pod "maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" (UID: "1dfe2a94-f678-4802-a954-6d454dab0f5e") : secret "maas-api-e2e-api-a-52abd4-serving-cert" not found Jul 20 03:27:58.099201 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.099179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.106707 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.106672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbb6\" (UniqueName: \"kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.602987 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.602950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.605374 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.605341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") pod \"maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.763317 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.763266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:58.909249 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.909215 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:27:58.911249 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:27:58.911220 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfe2a94_f678_4802_a954_6d454dab0f5e.slice/crio-93017c18ce89de2ce21024ade16f77a48af29954847b7e0c1f0e00e1bdeb4eb0 WatchSource:0}: Error finding container 93017c18ce89de2ce21024ade16f77a48af29954847b7e0c1f0e00e1bdeb4eb0: Status 404 returned error can't find the container with id 93017c18ce89de2ce21024ade16f77a48af29954847b7e0c1f0e00e1bdeb4eb0 Jul 20 03:27:58.912958 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:58.912941 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:27:59.620626 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:59.620587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" event={"ID":"1dfe2a94-f678-4802-a954-6d454dab0f5e","Type":"ContainerStarted","Data":"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f"} Jul 20 03:27:59.620812 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:59.620723 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:27:59.620812 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:59.620741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" event={"ID":"1dfe2a94-f678-4802-a954-6d454dab0f5e","Type":"ContainerStarted","Data":"93017c18ce89de2ce21024ade16f77a48af29954847b7e0c1f0e00e1bdeb4eb0"} Jul 20 03:27:59.637949 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:27:59.637903 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" podStartSLOduration=2.24142106 podStartE2EDuration="2.637887773s" totalImagesPullingTime="396.466713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:27:59.63559326 +0000 UTC m=+2574.511394983" watchObservedRunningTime="2026-07-20 03:27:59.637887773 +0000 UTC m=+2574.513689483" Jul 20 03:28:05.631028 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.631000 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:28:05.932248 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.932160 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:28:05.936016 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.935993 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:05.938139 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.938117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a8945e-serving-cert\"" Jul 20 03:28:05.947286 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.947259 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:28:05.970352 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.970312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66fk\" (UniqueName: \"kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:05.970486 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.970375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:05.970486 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:05.970410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.071141 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.071105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66fk\" (UniqueName: \"kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.071141 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.071146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.071415 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.071179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.071415 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:28:06.071270 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-serving-cert: secret "maas-api-e2e-api-b-a8945e-serving-cert" not found Jul 20 03:28:06.071415 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:28:06.071365 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls podName:8fe6eb15-731c-495a-808e-4dce838b4364 nodeName:}" failed. No retries permitted until 2026-07-20 03:28:06.571343651 +0000 UTC m=+2581.447145341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls") pod "maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" (UID: "8fe6eb15-731c-495a-808e-4dce838b4364") : secret "maas-api-e2e-api-b-a8945e-serving-cert" not found Jul 20 03:28:06.072035 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.072010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.079651 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.079622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66fk\" (UniqueName: \"kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.575389 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:06.575358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:06.575565 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:28:06.575547 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-serving-cert: secret "maas-api-e2e-api-b-a8945e-serving-cert" not found Jul 20 03:28:06.575653 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:28:06.575641 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls podName:8fe6eb15-731c-495a-808e-4dce838b4364 nodeName:}" failed. No retries permitted until 2026-07-20 03:28:07.575620586 +0000 UTC m=+2582.451422279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls") pod "maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" (UID: "8fe6eb15-731c-495a-808e-4dce838b4364") : secret "maas-api-e2e-api-b-a8945e-serving-cert" not found Jul 20 03:28:07.584713 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:07.584680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:07.587169 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:07.587137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:07.749489 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:07.749397 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:07.889601 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:07.889573 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:28:07.890666 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:28:07.890637 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe6eb15_731c_495a_808e_4dce838b4364.slice/crio-c19f2a31d108c751d66c11d569f5d06f5e86f91cf0d7b8660f0e685c183aece4 WatchSource:0}: Error finding container c19f2a31d108c751d66c11d569f5d06f5e86f91cf0d7b8660f0e685c183aece4: Status 404 returned error can't find the container with id c19f2a31d108c751d66c11d569f5d06f5e86f91cf0d7b8660f0e685c183aece4 Jul 20 03:28:08.673100 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:08.673066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" event={"ID":"8fe6eb15-731c-495a-808e-4dce838b4364","Type":"ContainerStarted","Data":"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179"} Jul 20 03:28:08.673555 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:08.673119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" event={"ID":"8fe6eb15-731c-495a-808e-4dce838b4364","Type":"ContainerStarted","Data":"c19f2a31d108c751d66c11d569f5d06f5e86f91cf0d7b8660f0e685c183aece4"} Jul 20 03:28:08.673555 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:08.673180 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:08.690181 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:08.690134 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" podStartSLOduration=3.271585912 podStartE2EDuration="3.690117431s" totalImagesPullingTime="418.531519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:28:08.686850641 +0000 UTC m=+2583.562652350" watchObservedRunningTime="2026-07-20 03:28:08.690117431 +0000 UTC m=+2583.565919142" Jul 20 03:28:14.684718 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:14.684685 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:28:34.147094 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.147003 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:28:34.147577 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.147260 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" podUID="1dfe2a94-f678-4802-a954-6d454dab0f5e" containerName="maas-api" containerID="cri-o://c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f" gracePeriod=30 Jul 20 03:28:34.404749 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.404691 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:28:34.551433 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.551399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca\") pod \"1dfe2a94-f678-4802-a954-6d454dab0f5e\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " Jul 20 03:28:34.551596 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.551451 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") pod \"1dfe2a94-f678-4802-a954-6d454dab0f5e\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " Jul 20 03:28:34.551596 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.551492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bbb6\" (UniqueName: \"kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6\") pod \"1dfe2a94-f678-4802-a954-6d454dab0f5e\" (UID: \"1dfe2a94-f678-4802-a954-6d454dab0f5e\") " Jul 20 03:28:34.551873 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.551841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1dfe2a94-f678-4802-a954-6d454dab0f5e" (UID: "1dfe2a94-f678-4802-a954-6d454dab0f5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:28:34.553621 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.553602 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1dfe2a94-f678-4802-a954-6d454dab0f5e" (UID: "1dfe2a94-f678-4802-a954-6d454dab0f5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:28:34.553697 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.553640 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6" (OuterVolumeSpecName: "kube-api-access-9bbb6") pod "1dfe2a94-f678-4802-a954-6d454dab0f5e" (UID: "1dfe2a94-f678-4802-a954-6d454dab0f5e"). InnerVolumeSpecName "kube-api-access-9bbb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:28:34.653167 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.653133 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfe2a94-f678-4802-a954-6d454dab0f5e-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:28:34.653167 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.653161 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfe2a94-f678-4802-a954-6d454dab0f5e-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:28:34.653167 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.653172 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bbb6\" (UniqueName: \"kubernetes.io/projected/1dfe2a94-f678-4802-a954-6d454dab0f5e-kube-api-access-9bbb6\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:28:34.822544 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.822454 2565 generic.go:356] "Generic (PLEG): container finished" podID="1dfe2a94-f678-4802-a954-6d454dab0f5e" containerID="c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f" exitCode=0 Jul 20 03:28:34.822544 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.822495 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" Jul 20 03:28:34.822751 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.822546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" event={"ID":"1dfe2a94-f678-4802-a954-6d454dab0f5e","Type":"ContainerDied","Data":"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f"} Jul 20 03:28:34.822751 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.822600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg" event={"ID":"1dfe2a94-f678-4802-a954-6d454dab0f5e","Type":"ContainerDied","Data":"93017c18ce89de2ce21024ade16f77a48af29954847b7e0c1f0e00e1bdeb4eb0"} Jul 20 03:28:34.823467 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.822627 2565 scope.go:117] "RemoveContainer" containerID="c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f" Jul 20 03:28:34.832606 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.832587 2565 scope.go:117] "RemoveContainer" containerID="c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f" Jul 20 03:28:34.832909 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:28:34.832873 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f\": container with ID starting with c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f not found: ID does not exist" containerID="c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f" Jul 20 03:28:34.832993 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.832918 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f"} err="failed to get container status \"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f\": rpc error: code = NotFound desc = could not find container \"c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f\": container with ID starting with c88d808236559514ca077a71ce3904e048f19e3c2dc75b15265eaa4eb089166f not found: ID does not exist" Jul 20 03:28:34.843264 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.843244 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:28:34.848114 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:34.848090 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-52abd4-584f8b548c-xgvjg"] Jul 20 03:28:35.192551 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:35.192520 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dfe2a94-f678-4802-a954-6d454dab0f5e" path="/var/lib/kubelet/pods/1dfe2a94-f678-4802-a954-6d454dab0f5e/volumes" Jul 20 03:28:44.980000 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.979965 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv"] Jul 20 03:28:44.980503 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.980484 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dfe2a94-f678-4802-a954-6d454dab0f5e" containerName="maas-api" Jul 20 03:28:44.980545 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.980505 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dfe2a94-f678-4802-a954-6d454dab0f5e" containerName="maas-api" Jul 20 03:28:44.980593 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.980583 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dfe2a94-f678-4802-a954-6d454dab0f5e" containerName="maas-api" Jul 20 03:28:44.983692 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.983674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:44.985631 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.985606 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:28:44.990359 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:44.990332 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv"] Jul 20 03:28:45.044900 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.044864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a8945e-lccbv\" (UID: \"4eae8878-fae6-47f3-ba97-e53e661aab41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:45.145532 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.145497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a8945e-lccbv\" (UID: \"4eae8878-fae6-47f3-ba97-e53e661aab41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:45.146145 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.146126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a8945e-lccbv\" (UID: \"4eae8878-fae6-47f3-ba97-e53e661aab41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:45.298332 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.298229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:45.438576 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.438551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv"] Jul 20 03:28:45.440510 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:28:45.440482 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eae8878_fae6_47f3_ba97_e53e661aab41.slice/crio-79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8 WatchSource:0}: Error finding container 79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8: Status 404 returned error can't find the container with id 79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8 Jul 20 03:28:45.896134 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.896094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" event={"ID":"4eae8878-fae6-47f3-ba97-e53e661aab41","Type":"ContainerStarted","Data":"4372fdec3ba746149ce5a422a71205ec1ec3045efd04455115b68d2f63850627"} Jul 20 03:28:45.896425 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.896184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" event={"ID":"4eae8878-fae6-47f3-ba97-e53e661aab41","Type":"ContainerStarted","Data":"79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8"} Jul 20 03:28:45.912461 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:45.912404 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" podStartSLOduration=1.912387099 podStartE2EDuration="1.912387099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:28:45.909838701 +0000 UTC m=+2620.785640416" watchObservedRunningTime="2026-07-20 03:28:45.912387099 +0000 UTC m=+2620.788188808" Jul 20 03:28:46.903303 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:46.903260 2565 generic.go:356] "Generic (PLEG): container finished" podID="4eae8878-fae6-47f3-ba97-e53e661aab41" containerID="4372fdec3ba746149ce5a422a71205ec1ec3045efd04455115b68d2f63850627" exitCode=0 Jul 20 03:28:46.903706 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:46.903347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" event={"ID":"4eae8878-fae6-47f3-ba97-e53e661aab41","Type":"ContainerDied","Data":"4372fdec3ba746149ce5a422a71205ec1ec3045efd04455115b68d2f63850627"} Jul 20 03:28:48.064459 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.064437 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:48.069354 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.069327 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca\") pod \"4eae8878-fae6-47f3-ba97-e53e661aab41\" (UID: \"4eae8878-fae6-47f3-ba97-e53e661aab41\") " Jul 20 03:28:48.069610 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.069587 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4eae8878-fae6-47f3-ba97-e53e661aab41" (UID: "4eae8878-fae6-47f3-ba97-e53e661aab41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:28:48.170402 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.170316 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eae8878-fae6-47f3-ba97-e53e661aab41-maas-api-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:28:48.916629 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.916601 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" Jul 20 03:28:48.916789 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.916594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv" event={"ID":"4eae8878-fae6-47f3-ba97-e53e661aab41","Type":"ContainerDied","Data":"79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8"} Jul 20 03:28:48.916789 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:28:48.916727 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d9b081a9569c9d0630d2ef466a09b877b569219055c03194aa7adb8e7fd0c8" Jul 20 03:29:00.400444 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.400408 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:29:00.400939 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.400682 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" podUID="8fe6eb15-731c-495a-808e-4dce838b4364" containerName="maas-api" containerID="cri-o://81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179" gracePeriod=30 Jul 20 03:29:00.655589 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.655525 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:29:00.788066 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.788034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") pod \"8fe6eb15-731c-495a-808e-4dce838b4364\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " Jul 20 03:29:00.788238 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.788107 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c66fk\" (UniqueName: \"kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk\") pod \"8fe6eb15-731c-495a-808e-4dce838b4364\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " Jul 20 03:29:00.788238 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.788141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca\") pod \"8fe6eb15-731c-495a-808e-4dce838b4364\" (UID: \"8fe6eb15-731c-495a-808e-4dce838b4364\") " Jul 20 03:29:00.788541 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.788501 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fe6eb15-731c-495a-808e-4dce838b4364" (UID: "8fe6eb15-731c-495a-808e-4dce838b4364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:29:00.790160 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.790137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fe6eb15-731c-495a-808e-4dce838b4364" (UID: "8fe6eb15-731c-495a-808e-4dce838b4364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:29:00.790263 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.790192 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk" (OuterVolumeSpecName: "kube-api-access-c66fk") pod "8fe6eb15-731c-495a-808e-4dce838b4364" (UID: "8fe6eb15-731c-495a-808e-4dce838b4364"). InnerVolumeSpecName "kube-api-access-c66fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:29:00.889758 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.889723 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe6eb15-731c-495a-808e-4dce838b4364-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:29:00.889758 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.889752 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66fk\" (UniqueName: \"kubernetes.io/projected/8fe6eb15-731c-495a-808e-4dce838b4364-kube-api-access-c66fk\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:29:00.889758 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.889762 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6eb15-731c-495a-808e-4dce838b4364-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:29:00.985119 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.985029 2565 generic.go:356] "Generic (PLEG): container finished" podID="8fe6eb15-731c-495a-808e-4dce838b4364" containerID="81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179" exitCode=0 Jul 20 03:29:00.985119 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.985073 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" Jul 20 03:29:00.985366 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.985122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" event={"ID":"8fe6eb15-731c-495a-808e-4dce838b4364","Type":"ContainerDied","Data":"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179"} Jul 20 03:29:00.985366 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.985181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk" event={"ID":"8fe6eb15-731c-495a-808e-4dce838b4364","Type":"ContainerDied","Data":"c19f2a31d108c751d66c11d569f5d06f5e86f91cf0d7b8660f0e685c183aece4"} Jul 20 03:29:00.985366 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.985198 2565 scope.go:117] "RemoveContainer" containerID="81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179" Jul 20 03:29:00.995158 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.995139 2565 scope.go:117] "RemoveContainer" containerID="81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179" Jul 20 03:29:00.995436 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:29:00.995415 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179\": container with ID starting with 81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179 not found: ID does not exist" containerID="81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179" Jul 20 03:29:00.995487 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:00.995444 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179"} err="failed to get container status \"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179\": rpc error: code = NotFound desc = could not find container \"81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179\": container with ID starting with 81acf204db9713b846af66888aada5bbb034a11fd2a626fef8bd45605fea7179 not found: ID does not exist" Jul 20 03:29:01.005782 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:01.005758 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:29:01.010626 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:01.010605 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a8945e-9cc66447d-5ffkk"] Jul 20 03:29:01.191738 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:01.191702 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe6eb15-731c-495a-808e-4dce838b4364" path="/var/lib/kubelet/pods/8fe6eb15-731c-495a-808e-4dce838b4364/volumes" Jul 20 03:29:05.059060 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:05.059021 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv"] Jul 20 03:29:05.063537 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:05.063511 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a8945e-lccbv"] Jul 20 03:29:05.191527 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:05.191489 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eae8878-fae6-47f3-ba97-e53e661aab41" path="/var/lib/kubelet/pods/4eae8878-fae6-47f3-ba97-e53e661aab41/volumes" Jul 20 03:29:15.233474 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233437 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233825 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe6eb15-731c-495a-808e-4dce838b4364" containerName="maas-api" Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233836 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe6eb15-731c-495a-808e-4dce838b4364" containerName="maas-api" Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233854 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eae8878-fae6-47f3-ba97-e53e661aab41" containerName="revoke-keys" Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233860 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eae8878-fae6-47f3-ba97-e53e661aab41" containerName="revoke-keys" Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233918 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe6eb15-731c-495a-808e-4dce838b4364" containerName="maas-api" Jul 20 03:29:15.233976 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.233926 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eae8878-fae6-47f3-ba97-e53e661aab41" containerName="revoke-keys" Jul 20 03:29:15.237243 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.237223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.239199 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.239179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-496193-serving-cert\"" Jul 20 03:29:15.249024 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.248997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:29:15.331978 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.331936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8nc\" (UniqueName: \"kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.332185 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.332030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.332185 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.332077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.433427 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.433390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.433607 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.433443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.433607 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.433516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8nc\" (UniqueName: \"kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.433680 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:29:15.433632 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-serving-cert: secret "maas-api-e2e-shared-a-496193-serving-cert" not found Jul 20 03:29:15.433715 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:29:15.433710 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls podName:16899373-8f52-48ac-ba86-a86303d4705d nodeName:}" failed. No retries permitted until 2026-07-20 03:29:15.933692621 +0000 UTC m=+2650.809494313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls") pod "maas-api-e2e-shared-a-496193-68694f748b-fgf9h" (UID: "16899373-8f52-48ac-ba86-a86303d4705d") : secret "maas-api-e2e-shared-a-496193-serving-cert" not found Jul 20 03:29:15.434143 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.434124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.444479 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.444449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8nc\" (UniqueName: \"kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.939136 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.939096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:15.941582 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:15.941553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") pod \"maas-api-e2e-shared-a-496193-68694f748b-fgf9h\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:16.149995 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:16.149960 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:16.297564 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:16.297528 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:29:16.299171 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:29:16.299141 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16899373_8f52_48ac_ba86_a86303d4705d.slice/crio-12c610b0ac4645794ec829fd02df22f1c58f50667cb92c6d7bc8dcb611e78c67 WatchSource:0}: Error finding container 12c610b0ac4645794ec829fd02df22f1c58f50667cb92c6d7bc8dcb611e78c67: Status 404 returned error can't find the container with id 12c610b0ac4645794ec829fd02df22f1c58f50667cb92c6d7bc8dcb611e78c67 Jul 20 03:29:17.078399 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:17.078362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" event={"ID":"16899373-8f52-48ac-ba86-a86303d4705d","Type":"ContainerStarted","Data":"f9aa9061d5a5013503f455ddcb2fbc0d969a08d61c4ace96ac790ecb6ffed3db"} Jul 20 03:29:17.078652 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:17.078425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:17.078652 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:17.078446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" event={"ID":"16899373-8f52-48ac-ba86-a86303d4705d","Type":"ContainerStarted","Data":"12c610b0ac4645794ec829fd02df22f1c58f50667cb92c6d7bc8dcb611e78c67"} Jul 20 03:29:17.099606 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:17.099547 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" podStartSLOduration=1.66864811 podStartE2EDuration="2.099531958s" totalImagesPullingTime="430.883848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:29:17.097254629 +0000 UTC m=+2651.973056340" watchObservedRunningTime="2026-07-20 03:29:17.099531958 +0000 UTC m=+2651.975333689" Jul 20 03:29:23.090225 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.090193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:29:23.245176 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.245143 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:29:23.249183 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.249160 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.252052 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.252030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c1951c-serving-cert\"" Jul 20 03:29:23.263392 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.263364 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:29:23.412837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.412732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.412837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.412813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhct\" (UniqueName: \"kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.413073 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.412922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.513570 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.513529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.513771 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.513583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.513771 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.513638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhct\" (UniqueName: \"kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.513771 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:29:23.513674 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-serving-cert: secret "maas-api-e2e-shared-b-c1951c-serving-cert" not found Jul 20 03:29:23.513771 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:29:23.513736 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls podName:655a6a1a-c170-43be-ad24-6bcf14a5577d nodeName:}" failed. No retries permitted until 2026-07-20 03:29:24.013718727 +0000 UTC m=+2658.889520419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls") pod "maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" (UID: "655a6a1a-c170-43be-ad24-6bcf14a5577d") : secret "maas-api-e2e-shared-b-c1951c-serving-cert" not found Jul 20 03:29:23.514331 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.514287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:23.521880 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:23.521855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhct\" (UniqueName: \"kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:24.018703 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:24.018664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:24.021249 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:24.021219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:24.161278 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:24.161241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:24.313537 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:24.313501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:29:24.314486 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:29:24.314458 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655a6a1a_c170_43be_ad24_6bcf14a5577d.slice/crio-4555f5ae139fac086902f5a35ff5dff12741a69d62aa1df4a60b4d428f313ca7 WatchSource:0}: Error finding container 4555f5ae139fac086902f5a35ff5dff12741a69d62aa1df4a60b4d428f313ca7: Status 404 returned error can't find the container with id 4555f5ae139fac086902f5a35ff5dff12741a69d62aa1df4a60b4d428f313ca7 Jul 20 03:29:25.125519 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:25.125483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" event={"ID":"655a6a1a-c170-43be-ad24-6bcf14a5577d","Type":"ContainerStarted","Data":"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a"} Jul 20 03:29:25.125699 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:25.125544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" event={"ID":"655a6a1a-c170-43be-ad24-6bcf14a5577d","Type":"ContainerStarted","Data":"4555f5ae139fac086902f5a35ff5dff12741a69d62aa1df4a60b4d428f313ca7"} Jul 20 03:29:25.125699 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:25.125574 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:29:25.141798 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:25.141750 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" podStartSLOduration=1.797736091 podStartE2EDuration="2.141735367s" totalImagesPullingTime="343.999276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:29:25.139767377 +0000 UTC m=+2660.015569087" watchObservedRunningTime="2026-07-20 03:29:25.141735367 +0000 UTC m=+2660.017537077" Jul 20 03:29:31.137367 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:29:31.137339 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:30:07.240589 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:30:07.240510 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:30:07.247179 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:30:07.247156 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:30:07.249227 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:30:07.249208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:30:07.255601 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:30:07.255584 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:35:07.277735 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:07.277700 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:35:07.284786 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:07.284759 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:35:07.287897 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:07.287872 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:35:07.294990 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:07.294970 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:35:07.534765 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:07.534688 2565 scope.go:117] "RemoveContainer" containerID="4372fdec3ba746149ce5a422a71205ec1ec3045efd04455115b68d2f63850627" Jul 20 03:35:13.057022 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.056984 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7313-139914754126128"] Jul 20 03:35:13.061957 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.061925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:13.063970 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.063945 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kg5k\"" Jul 20 03:35:13.065576 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.065548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754126128"] Jul 20 03:35:13.137225 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.137180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq5s\" (UniqueName: \"kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s\") pod \"test-curl-7313-139914754126128\" (UID: \"996780e2-cd7d-48b4-85b7-de3cc83f82b9\") " pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:13.237633 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.237594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq5s\" (UniqueName: \"kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s\") pod \"test-curl-7313-139914754126128\" (UID: \"996780e2-cd7d-48b4-85b7-de3cc83f82b9\") " pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:13.245772 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.245735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq5s\" (UniqueName: \"kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s\") pod \"test-curl-7313-139914754126128\" (UID: \"996780e2-cd7d-48b4-85b7-de3cc83f82b9\") " pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:13.373668 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.373630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:13.509276 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.509246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754126128"] Jul 20 03:35:13.511085 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:35:13.511059 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod996780e2_cd7d_48b4_85b7_de3cc83f82b9.slice/crio-95d49fff541146e37f071ca33364502f217a080690e03ea6feb0048c317bec9f WatchSource:0}: Error finding container 95d49fff541146e37f071ca33364502f217a080690e03ea6feb0048c317bec9f: Status 404 returned error can't find the container with id 95d49fff541146e37f071ca33364502f217a080690e03ea6feb0048c317bec9f Jul 20 03:35:13.512567 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:13.512545 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:35:14.109751 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:14.109714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754126128" event={"ID":"996780e2-cd7d-48b4-85b7-de3cc83f82b9","Type":"ContainerStarted","Data":"95d49fff541146e37f071ca33364502f217a080690e03ea6feb0048c317bec9f"} Jul 20 03:35:15.117110 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:15.117075 2565 generic.go:356] "Generic (PLEG): container finished" podID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" containerID="4ea6a58e7b65154e13cc92a24fb08bfcb70cc0bc82d08cc15439eff3b43e778f" exitCode=0 Jul 20 03:35:15.117599 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:15.117158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754126128" event={"ID":"996780e2-cd7d-48b4-85b7-de3cc83f82b9","Type":"ContainerDied","Data":"4ea6a58e7b65154e13cc92a24fb08bfcb70cc0bc82d08cc15439eff3b43e778f"} Jul 20 03:35:16.260004 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.259978 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:16.300382 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.300282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7313-139914754126128_996780e2-cd7d-48b4-85b7-de3cc83f82b9/test-curl-7313-139914754126128/0.log" Jul 20 03:35:16.332506 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.332471 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7313-139914754126128"] Jul 20 03:35:16.337025 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.336994 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7313-139914754126128"] Jul 20 03:35:16.363484 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.363459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzq5s\" (UniqueName: \"kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s\") pod \"996780e2-cd7d-48b4-85b7-de3cc83f82b9\" (UID: \"996780e2-cd7d-48b4-85b7-de3cc83f82b9\") " Jul 20 03:35:16.365603 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.365578 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s" (OuterVolumeSpecName: "kube-api-access-mzq5s") pod "996780e2-cd7d-48b4-85b7-de3cc83f82b9" (UID: "996780e2-cd7d-48b4-85b7-de3cc83f82b9"). InnerVolumeSpecName "kube-api-access-mzq5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:16.464493 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:16.464465 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq5s\" (UniqueName: \"kubernetes.io/projected/996780e2-cd7d-48b4-85b7-de3cc83f82b9-kube-api-access-mzq5s\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:17.130053 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:17.130026 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:17.130053 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:17.130041 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d49fff541146e37f071ca33364502f217a080690e03ea6feb0048c317bec9f" Jul 20 03:35:17.139726 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:35:17.139697 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7313-139914754126128\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:17.191888 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:35:17.191852 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7313-139914754126128\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" pod="opendatahub/test-curl-7313-139914754126128" Jul 20 03:35:17.192020 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:17.191945 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" path="/var/lib/kubelet/pods/996780e2-cd7d-48b4-85b7-de3cc83f82b9/volumes" Jul 20 03:35:22.102591 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.102554 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:22.102977 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.102956 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" containerName="test-curl-7313-139914754126128" Jul 20 03:35:22.102977 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.102968 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" containerName="test-curl-7313-139914754126128" Jul 20 03:35:22.103047 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.103030 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="996780e2-cd7d-48b4-85b7-de3cc83f82b9" containerName="test-curl-7313-139914754126128" Jul 20 03:35:22.106404 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.106379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:22.108429 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.108401 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8kg5k\"" Jul 20 03:35:22.110098 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.110071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:22.222513 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.222477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbw5\" (UniqueName: \"kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5\") pod \"test-curl-7313-139914754214048\" (UID: \"9d368a85-5c33-44d3-8585-69a501e0599c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:22.323504 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.323471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbw5\" (UniqueName: \"kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5\") pod \"test-curl-7313-139914754214048\" (UID: \"9d368a85-5c33-44d3-8585-69a501e0599c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:22.332132 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.332106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbw5\" (UniqueName: \"kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5\") pod \"test-curl-7313-139914754214048\" (UID: \"9d368a85-5c33-44d3-8585-69a501e0599c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:22.417123 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.417043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:22.551808 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:22.551777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:22.553334 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:35:22.553284 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d368a85_5c33_44d3_8585_69a501e0599c.slice/crio-a0a1dc6ce35967f919d672189b45536caeeb77b8a2c923fb2fb987d44ee1538b WatchSource:0}: Error finding container a0a1dc6ce35967f919d672189b45536caeeb77b8a2c923fb2fb987d44ee1538b: Status 404 returned error can't find the container with id a0a1dc6ce35967f919d672189b45536caeeb77b8a2c923fb2fb987d44ee1538b Jul 20 03:35:23.164838 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:23.164739 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d368a85-5c33-44d3-8585-69a501e0599c" containerID="70915e3311021ed18e03a8a1296ed659039e01398ef40e4187a1274d346e624f" exitCode=0 Jul 20 03:35:23.165189 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:23.164832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754214048" event={"ID":"9d368a85-5c33-44d3-8585-69a501e0599c","Type":"ContainerDied","Data":"70915e3311021ed18e03a8a1296ed659039e01398ef40e4187a1274d346e624f"} Jul 20 03:35:23.165189 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:23.164873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754214048" event={"ID":"9d368a85-5c33-44d3-8585-69a501e0599c","Type":"ContainerStarted","Data":"a0a1dc6ce35967f919d672189b45536caeeb77b8a2c923fb2fb987d44ee1538b"} Jul 20 03:35:24.304367 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.304344 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.326788 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.326762 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7313-139914754214048_9d368a85-5c33-44d3-8585-69a501e0599c/test-curl-7313-139914754214048/0.log" Jul 20 03:35:24.344054 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.344024 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbw5\" (UniqueName: \"kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5\") pod \"9d368a85-5c33-44d3-8585-69a501e0599c\" (UID: \"9d368a85-5c33-44d3-8585-69a501e0599c\") " Jul 20 03:35:24.346038 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.346010 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5" (OuterVolumeSpecName: "kube-api-access-tlbw5") pod "9d368a85-5c33-44d3-8585-69a501e0599c" (UID: "9d368a85-5c33-44d3-8585-69a501e0599c"). InnerVolumeSpecName "kube-api-access-tlbw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:24.352228 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.352204 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:24.356049 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.356025 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:24.445666 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.445590 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbw5\" (UniqueName: \"kubernetes.io/projected/9d368a85-5c33-44d3-8585-69a501e0599c-kube-api-access-tlbw5\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:24.476963 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.476933 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:24.477442 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.477427 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d368a85-5c33-44d3-8585-69a501e0599c" containerName="test-curl-7313-139914754214048" Jul 20 03:35:24.477490 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.477444 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d368a85-5c33-44d3-8585-69a501e0599c" containerName="test-curl-7313-139914754214048" Jul 20 03:35:24.477531 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.477504 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d368a85-5c33-44d3-8585-69a501e0599c" containerName="test-curl-7313-139914754214048" Jul 20 03:35:24.480517 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.480498 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.485066 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.485037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:24.547100 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.547067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgr6\" (UniqueName: \"kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6\") pod \"test-curl-7313-139914754214048\" (UID: \"77f7b55a-a455-440e-b05b-901c8b9de62c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.648335 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.648271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgr6\" (UniqueName: \"kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6\") pod \"test-curl-7313-139914754214048\" (UID: \"77f7b55a-a455-440e-b05b-901c8b9de62c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.657705 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.657655 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgr6\" (UniqueName: \"kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6\") pod \"test-curl-7313-139914754214048\" (UID: \"77f7b55a-a455-440e-b05b-901c8b9de62c\") " pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.792743 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.792649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:24.925802 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:24.925768 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:24.929261 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:35:24.929235 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f7b55a_a455_440e_b05b_901c8b9de62c.slice/crio-1673eaf0202a5cd42ef3dba115a4338899cb6574770dc118962afcfcdbae7ca3 WatchSource:0}: Error finding container 1673eaf0202a5cd42ef3dba115a4338899cb6574770dc118962afcfcdbae7ca3: Status 404 returned error can't find the container with id 1673eaf0202a5cd42ef3dba115a4338899cb6574770dc118962afcfcdbae7ca3 Jul 20 03:35:25.180125 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:25.180079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754214048" event={"ID":"77f7b55a-a455-440e-b05b-901c8b9de62c","Type":"ContainerStarted","Data":"1673eaf0202a5cd42ef3dba115a4338899cb6574770dc118962afcfcdbae7ca3"} Jul 20 03:35:25.181426 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:25.181397 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a1dc6ce35967f919d672189b45536caeeb77b8a2c923fb2fb987d44ee1538b" Jul 20 03:35:25.181426 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:25.181407 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:25.192956 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:25.192932 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d368a85-5c33-44d3-8585-69a501e0599c" path="/var/lib/kubelet/pods/9d368a85-5c33-44d3-8585-69a501e0599c/volumes" Jul 20 03:35:25.193202 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:25.193176 2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7313-139914754214048" oldPodUID="9d368a85-5c33-44d3-8585-69a501e0599c" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" Jul 20 03:35:26.188433 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:26.188396 2565 generic.go:356] "Generic (PLEG): container finished" podID="77f7b55a-a455-440e-b05b-901c8b9de62c" containerID="c14af7d83af06ac902f430266dc63b080715007de8bd476490bff444eca90f70" exitCode=0 Jul 20 03:35:26.188820 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:26.188482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7313-139914754214048" event={"ID":"77f7b55a-a455-440e-b05b-901c8b9de62c","Type":"ContainerDied","Data":"c14af7d83af06ac902f430266dc63b080715007de8bd476490bff444eca90f70"} Jul 20 03:35:27.329254 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.329227 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:27.349059 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.349035 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7313-139914754214048_77f7b55a-a455-440e-b05b-901c8b9de62c/test-curl-7313-139914754214048/0.log" Jul 20 03:35:27.373973 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.373933 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:27.376304 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.376268 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7313-139914754214048"] Jul 20 03:35:27.377527 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.377502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfgr6\" (UniqueName: \"kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6\") pod \"77f7b55a-a455-440e-b05b-901c8b9de62c\" (UID: \"77f7b55a-a455-440e-b05b-901c8b9de62c\") " Jul 20 03:35:27.379530 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.379496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6" (OuterVolumeSpecName: "kube-api-access-hfgr6") pod "77f7b55a-a455-440e-b05b-901c8b9de62c" (UID: "77f7b55a-a455-440e-b05b-901c8b9de62c"). InnerVolumeSpecName "kube-api-access-hfgr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:27.478850 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:27.478763 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfgr6\" (UniqueName: \"kubernetes.io/projected/77f7b55a-a455-440e-b05b-901c8b9de62c-kube-api-access-hfgr6\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:28.199999 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:28.199966 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1673eaf0202a5cd42ef3dba115a4338899cb6574770dc118962afcfcdbae7ca3" Jul 20 03:35:28.199999 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:28.199987 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:28.209723 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:35:28.209693 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7313-139914754214048\" is forbidden: User \"system:node:ip-10-0-134-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-38.ec2.internal' and this object" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" pod="opendatahub/test-curl-7313-139914754214048" Jul 20 03:35:29.191725 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:29.191692 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" path="/var/lib/kubelet/pods/77f7b55a-a455-440e-b05b-901c8b9de62c/volumes" Jul 20 03:35:38.102032 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.101983 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:35:38.102667 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.102240 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" podUID="16899373-8f52-48ac-ba86-a86303d4705d" containerName="maas-api" containerID="cri-o://f9aa9061d5a5013503f455ddcb2fbc0d969a08d61c4ace96ac790ecb6ffed3db" gracePeriod=30 Jul 20 03:35:38.255264 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.255229 2565 generic.go:356] "Generic (PLEG): container finished" podID="16899373-8f52-48ac-ba86-a86303d4705d" containerID="f9aa9061d5a5013503f455ddcb2fbc0d969a08d61c4ace96ac790ecb6ffed3db" exitCode=0 Jul 20 03:35:38.255486 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.255322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" event={"ID":"16899373-8f52-48ac-ba86-a86303d4705d","Type":"ContainerDied","Data":"f9aa9061d5a5013503f455ddcb2fbc0d969a08d61c4ace96ac790ecb6ffed3db"} Jul 20 03:35:38.354621 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.354557 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:35:38.483577 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.483535 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") pod \"16899373-8f52-48ac-ba86-a86303d4705d\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " Jul 20 03:35:38.483780 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.483604 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b8nc\" (UniqueName: \"kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc\") pod \"16899373-8f52-48ac-ba86-a86303d4705d\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " Jul 20 03:35:38.483780 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.483628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca\") pod \"16899373-8f52-48ac-ba86-a86303d4705d\" (UID: \"16899373-8f52-48ac-ba86-a86303d4705d\") " Jul 20 03:35:38.483994 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.483971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16899373-8f52-48ac-ba86-a86303d4705d" (UID: "16899373-8f52-48ac-ba86-a86303d4705d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:35:38.485759 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.485735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16899373-8f52-48ac-ba86-a86303d4705d" (UID: "16899373-8f52-48ac-ba86-a86303d4705d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:35:38.485891 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.485871 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc" (OuterVolumeSpecName: "kube-api-access-2b8nc") pod "16899373-8f52-48ac-ba86-a86303d4705d" (UID: "16899373-8f52-48ac-ba86-a86303d4705d"). InnerVolumeSpecName "kube-api-access-2b8nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:38.585421 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.585387 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8nc\" (UniqueName: \"kubernetes.io/projected/16899373-8f52-48ac-ba86-a86303d4705d-kube-api-access-2b8nc\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:38.585421 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.585417 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16899373-8f52-48ac-ba86-a86303d4705d-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:38.585421 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:38.585427 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16899373-8f52-48ac-ba86-a86303d4705d-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:35:39.261902 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:39.261869 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" Jul 20 03:35:39.261902 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:39.261880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h" event={"ID":"16899373-8f52-48ac-ba86-a86303d4705d","Type":"ContainerDied","Data":"12c610b0ac4645794ec829fd02df22f1c58f50667cb92c6d7bc8dcb611e78c67"} Jul 20 03:35:39.262398 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:39.261943 2565 scope.go:117] "RemoveContainer" containerID="f9aa9061d5a5013503f455ddcb2fbc0d969a08d61c4ace96ac790ecb6ffed3db" Jul 20 03:35:39.278019 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:39.277994 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:35:39.281408 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:39.281384 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-496193-68694f748b-fgf9h"] Jul 20 03:35:41.192015 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:41.191982 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16899373-8f52-48ac-ba86-a86303d4705d" path="/var/lib/kubelet/pods/16899373-8f52-48ac-ba86-a86303d4705d/volumes" Jul 20 03:35:59.790205 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:59.790106 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:35:59.791037 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:35:59.790991 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" podUID="655a6a1a-c170-43be-ad24-6bcf14a5577d" containerName="maas-api" containerID="cri-o://527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a" gracePeriod=30 Jul 20 03:36:00.041258 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.041198 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:36:00.082974 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.082934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmhct\" (UniqueName: \"kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct\") pod \"655a6a1a-c170-43be-ad24-6bcf14a5577d\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " Jul 20 03:36:00.082974 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.082980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") pod \"655a6a1a-c170-43be-ad24-6bcf14a5577d\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " Jul 20 03:36:00.083230 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.083028 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca\") pod \"655a6a1a-c170-43be-ad24-6bcf14a5577d\" (UID: \"655a6a1a-c170-43be-ad24-6bcf14a5577d\") " Jul 20 03:36:00.083491 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.083460 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "655a6a1a-c170-43be-ad24-6bcf14a5577d" (UID: "655a6a1a-c170-43be-ad24-6bcf14a5577d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:36:00.085165 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.085135 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "655a6a1a-c170-43be-ad24-6bcf14a5577d" (UID: "655a6a1a-c170-43be-ad24-6bcf14a5577d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:36:00.085165 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.085155 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct" (OuterVolumeSpecName: "kube-api-access-xmhct") pod "655a6a1a-c170-43be-ad24-6bcf14a5577d" (UID: "655a6a1a-c170-43be-ad24-6bcf14a5577d"). InnerVolumeSpecName "kube-api-access-xmhct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:36:00.183940 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.183901 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhct\" (UniqueName: \"kubernetes.io/projected/655a6a1a-c170-43be-ad24-6bcf14a5577d-kube-api-access-xmhct\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:36:00.183940 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.183933 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655a6a1a-c170-43be-ad24-6bcf14a5577d-maas-api-tls\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:36:00.183940 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.183943 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655a6a1a-c170-43be-ad24-6bcf14a5577d-openshift-service-ca\") on node \"ip-10-0-134-38.ec2.internal\" DevicePath \"\"" Jul 20 03:36:00.384309 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.384263 2565 generic.go:356] "Generic (PLEG): container finished" podID="655a6a1a-c170-43be-ad24-6bcf14a5577d" containerID="527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a" exitCode=0 Jul 20 03:36:00.384464 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.384327 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" Jul 20 03:36:00.384464 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.384326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" event={"ID":"655a6a1a-c170-43be-ad24-6bcf14a5577d","Type":"ContainerDied","Data":"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a"} Jul 20 03:36:00.384464 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.384400 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs" event={"ID":"655a6a1a-c170-43be-ad24-6bcf14a5577d","Type":"ContainerDied","Data":"4555f5ae139fac086902f5a35ff5dff12741a69d62aa1df4a60b4d428f313ca7"} Jul 20 03:36:00.384464 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.384432 2565 scope.go:117] "RemoveContainer" containerID="527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a" Jul 20 03:36:00.394598 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.394582 2565 scope.go:117] "RemoveContainer" containerID="527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a" Jul 20 03:36:00.394854 ip-10-0-134-38 kubenswrapper[2565]: E0720 03:36:00.394834 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a\": container with ID starting with 527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a not found: ID does not exist" containerID="527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a" Jul 20 03:36:00.394910 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.394863 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a"} err="failed to get container status \"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a\": rpc error: code = NotFound desc = could not find container \"527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a\": container with ID starting with 527e3cd358a15502eb05105795d1b15c3d3a27d3277c424e60dbd6342c326d5a not found: ID does not exist" Jul 20 03:36:00.405750 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.405719 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:36:00.409412 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:00.409390 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c1951c-7657c79d7c-6f9cs"] Jul 20 03:36:01.192058 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:01.192024 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655a6a1a-c170-43be-ad24-6bcf14a5577d" path="/var/lib/kubelet/pods/655a6a1a-c170-43be-ad24-6bcf14a5577d/volumes" Jul 20 03:36:12.136901 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:12.136864 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544c679c6-2h7xn_2eb292be-3f9b-42a0-8faa-3968243d0f03/authorino/0.log" Jul 20 03:36:17.774263 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:17.774229 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pb9bv_1b6c66e2-5522-4894-a420-3e0381a84cbf/manager/0.log" Jul 20 03:36:17.898747 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:17.898717 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5msd8_abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d/manager/0.log" Jul 20 03:36:18.025085 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:18.025009 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645d8c8bc6-hxh5g_fda65796-3ea2-47ec-8b5f-b6db2b173d31/manager/0.log" Jul 20 03:36:18.387259 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:18.387230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb_11461e94-3b0d-4e4e-9137-995511b4ceb0/manager/0.log" Jul 20 03:36:18.393837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:18.393815 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb_11461e94-3b0d-4e4e-9137-995511b4ceb0/copy-manifests/0.log" Jul 20 03:36:19.051059 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:19.051019 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79b94c59c6-fzswd_c59ddee8-502d-4ee4-b037-d3b09a18062a/maas-api/0.log" Jul 20 03:36:19.168318 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:19.168270 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741940-x6vg6_2975a3da-4598-499b-b380-c917778d31d4/cleanup/0.log" Jul 20 03:36:19.496926 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:19.496893 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-td7k5_aa076322-2904-4691-93ba-20979ff3ea6f/postgres/0.log" Jul 20 03:36:20.323235 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.323203 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/util/0.log" Jul 20 03:36:20.330083 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.330049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/pull/0.log" Jul 20 03:36:20.336743 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.336723 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/extract/0.log" Jul 20 03:36:20.453850 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.453826 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/util/0.log" Jul 20 03:36:20.460591 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.460565 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/pull/0.log" Jul 20 03:36:20.467104 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.467087 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/extract/0.log" Jul 20 03:36:20.583672 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.583643 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/util/0.log" Jul 20 03:36:20.589846 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.589818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/pull/0.log" Jul 20 03:36:20.595813 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.595788 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/extract/0.log" Jul 20 03:36:20.706846 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.706817 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/util/0.log" Jul 20 03:36:20.713330 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.713306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/pull/0.log" Jul 20 03:36:20.719613 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.719592 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/extract/0.log" Jul 20 03:36:20.835248 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.835174 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544c679c6-2h7xn_2eb292be-3f9b-42a0-8faa-3968243d0f03/authorino/0.log" Jul 20 03:36:20.952241 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:20.952205 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sz6gh_1db3106f-5a3c-4ca2-a277-6ad3439e676b/manager/0.log" Jul 20 03:36:21.070063 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:21.070033 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8btkn_77435ef7-6ddc-4b38-9dde-78c52cf8767e/manager/0.log" Jul 20 03:36:21.182541 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:21.182463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jhn8c_26aa1c8f-e647-402d-8557-5cd153bfc98a/kuadrant-console-plugin/0.log" Jul 20 03:36:21.306827 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:21.306799 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fwqxc_cd8d2a4b-5afa-4939-b397-65b73b57840a/registry-server/0.log" Jul 20 03:36:21.428080 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:21.428046 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vqwhr_f8419e8d-8365-4e0f-aae0-e4441d115b95/manager/0.log" Jul 20 03:36:22.370274 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:22.370240 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d88568984-wwmc2_98682004-cb14-4a45-afea-0f41e3695832/kube-auth-proxy/0.log" Jul 20 03:36:22.629631 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:22.629555 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bltbd_65f000f6-0a19-4e6c-aded-34677ef8fc3d/payload-pre-processing/0.log" Jul 20 03:36:22.746254 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:22.746219 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hk2g6_773f6e47-f1d3-4555-8680-c23e0ce263c3/payload-processing/0.log" Jul 20 03:36:23.203347 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:23.203316 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-hl489_6b343014-6632-4b93-b7c3-ea62ae71e40a/main/0.log" Jul 20 03:36:23.317459 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:23.317427 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2m4nh_7a98653c-abbf-4d61-8214-4f78189b440b/main/0.log" Jul 20 03:36:23.663951 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:23.663920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cdq4f_54886eb1-f2b7-47be-bb68-bbad8d957bc8/main/0.log" Jul 20 03:36:23.780279 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:23.780246 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vtbrf_118eb4e8-4fec-41ce-97e5-d032ec7dc661/main/0.log" Jul 20 03:36:32.154817 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:32.154786 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mm7wt_2158b0e6-20c0-4fbf-9eb1-a45566f1b034/global-pull-secret-syncer/0.log" Jul 20 03:36:32.265047 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:32.265014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dv9tw_52576f0b-79a1-48bc-8438-ba81338d0429/konnectivity-agent/0.log" Jul 20 03:36:32.313728 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:32.313697 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-38.ec2.internal_73ae218cafc3c7d005fc46d807fa2771/haproxy/0.log" Jul 20 03:36:37.114707 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.114675 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/extract/0.log" Jul 20 03:36:37.135706 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.135677 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/util/0.log" Jul 20 03:36:37.156874 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.156846 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bmxkh_07fc172c-2df2-4131-8be7-17d294d0b13e/pull/0.log" Jul 20 03:36:37.184344 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.184315 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/extract/0.log" Jul 20 03:36:37.204413 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.204390 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/util/0.log" Jul 20 03:36:37.224473 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.224443 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04jwrc_d5c2b239-dc2f-4caa-9029-f47e09b1f32d/pull/0.log" Jul 20 03:36:37.249357 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.249333 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/extract/0.log" Jul 20 03:36:37.269564 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.269542 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/util/0.log" Jul 20 03:36:37.292577 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.292554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j5rbt_4270937e-3f48-42a7-a4d7-253dccb04da0/pull/0.log" Jul 20 03:36:37.323117 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.323085 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/extract/0.log" Jul 20 03:36:37.348048 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.348025 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/util/0.log" Jul 20 03:36:37.368714 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.368630 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hqbzq_80c4704e-51cd-482b-9207-781a3d502a9c/pull/0.log" Jul 20 03:36:37.702106 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.702021 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544c679c6-2h7xn_2eb292be-3f9b-42a0-8faa-3968243d0f03/authorino/0.log" Jul 20 03:36:37.732065 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.732037 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sz6gh_1db3106f-5a3c-4ca2-a277-6ad3439e676b/manager/0.log" Jul 20 03:36:37.759406 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.759379 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8btkn_77435ef7-6ddc-4b38-9dde-78c52cf8767e/manager/0.log" Jul 20 03:36:37.786100 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.786072 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jhn8c_26aa1c8f-e647-402d-8557-5cd153bfc98a/kuadrant-console-plugin/0.log" Jul 20 03:36:37.826010 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.825981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fwqxc_cd8d2a4b-5afa-4939-b397-65b73b57840a/registry-server/0.log" Jul 20 03:36:37.947232 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:37.947194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vqwhr_f8419e8d-8365-4e0f-aae0-e4441d115b95/manager/0.log" Jul 20 03:36:40.042475 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.042447 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-c8fk6_32784c61-0c3b-427c-9848-c2bfab6d7df4/kube-state-metrics/0.log" Jul 20 03:36:40.059401 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.059371 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-c8fk6_32784c61-0c3b-427c-9848-c2bfab6d7df4/kube-rbac-proxy-main/0.log" Jul 20 03:36:40.078070 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.078049 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-c8fk6_32784c61-0c3b-427c-9848-c2bfab6d7df4/kube-rbac-proxy-self/0.log" Jul 20 03:36:40.103327 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.103287 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b596fcdc4-q4cqr_b3b6a0ff-1c36-49fb-8ae7-f86ec51f488b/metrics-server/0.log" Jul 20 03:36:40.173270 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.173245 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvqct_c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2/node-exporter/0.log" Jul 20 03:36:40.193621 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.193603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvqct_c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2/kube-rbac-proxy/0.log" Jul 20 03:36:40.214105 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.214086 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvqct_c4b5cf75-76c5-4ea7-aa29-d328a32dd7a2/init-textfile/0.log" Jul 20 03:36:40.381939 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.381901 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cch6l_9324f46d-d7fd-4a95-9f49-b5dddca7cfde/kube-rbac-proxy-main/0.log" Jul 20 03:36:40.401116 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.401060 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cch6l_9324f46d-d7fd-4a95-9f49-b5dddca7cfde/kube-rbac-proxy-self/0.log" Jul 20 03:36:40.419950 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:40.419926 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cch6l_9324f46d-d7fd-4a95-9f49-b5dddca7cfde/openshift-state-metrics/0.log" Jul 20 03:36:42.225403 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225375 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd"] Jul 20 03:36:42.225798 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225776 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655a6a1a-c170-43be-ad24-6bcf14a5577d" containerName="maas-api" Jul 20 03:36:42.225798 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225788 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="655a6a1a-c170-43be-ad24-6bcf14a5577d" containerName="maas-api" Jul 20 03:36:42.225880 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225805 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16899373-8f52-48ac-ba86-a86303d4705d" containerName="maas-api" Jul 20 03:36:42.225880 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225814 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="16899373-8f52-48ac-ba86-a86303d4705d" containerName="maas-api" Jul 20 03:36:42.225880 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225830 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" containerName="test-curl-7313-139914754214048" Jul 20 03:36:42.225880 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225836 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" containerName="test-curl-7313-139914754214048" Jul 20 03:36:42.226013 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225899 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f7b55a-a455-440e-b05b-901c8b9de62c" containerName="test-curl-7313-139914754214048" Jul 20 03:36:42.226013 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225909 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="655a6a1a-c170-43be-ad24-6bcf14a5577d" containerName="maas-api" Jul 20 03:36:42.226013 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.225916 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="16899373-8f52-48ac-ba86-a86303d4705d" containerName="maas-api" Jul 20 03:36:42.229213 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.229197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.231310 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.231270 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spsmj\"/\"kube-root-ca.crt\"" Jul 20 03:36:42.231497 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.231483 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spsmj\"/\"openshift-service-ca.crt\"" Jul 20 03:36:42.231942 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.231931 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-spsmj\"/\"default-dockercfg-6h8fx\"" Jul 20 03:36:42.238796 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.238770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd"] Jul 20 03:36:42.371739 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.371694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-sys\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.371921 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.371818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-lib-modules\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.371921 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.371874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-podres\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.372022 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.371925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-proc\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.372022 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.372006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcc6h\" (UniqueName: \"kubernetes.io/projected/3536c191-9aec-4cc3-9b59-a34bacf23e48-kube-api-access-wcc6h\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472719 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-sys\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-lib-modules\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-podres\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-proc\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-sys\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcc6h\" (UniqueName: \"kubernetes.io/projected/3536c191-9aec-4cc3-9b59-a34bacf23e48-kube-api-access-wcc6h\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-podres\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-lib-modules\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.472917 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.472897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3536c191-9aec-4cc3-9b59-a34bacf23e48-proc\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.480598 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.480543 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcc6h\" (UniqueName: \"kubernetes.io/projected/3536c191-9aec-4cc3-9b59-a34bacf23e48-kube-api-access-wcc6h\") pod \"perf-node-gather-daemonset-4vdcd\" (UID: \"3536c191-9aec-4cc3-9b59-a34bacf23e48\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.540846 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.540818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:42.675138 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.675108 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd"] Jul 20 03:36:42.676556 ip-10-0-134-38 kubenswrapper[2565]: W0720 03:36:42.676527 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3536c191_9aec_4cc3_9b59_a34bacf23e48.slice/crio-e8b0bba4b9fe01978a6af1a4ee5a371c905111f04496e876ae04dcb2ee8a8ab7 WatchSource:0}: Error finding container e8b0bba4b9fe01978a6af1a4ee5a371c905111f04496e876ae04dcb2ee8a8ab7: Status 404 returned error can't find the container with id e8b0bba4b9fe01978a6af1a4ee5a371c905111f04496e876ae04dcb2ee8a8ab7 Jul 20 03:36:42.684802 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.684773 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/1.log" Jul 20 03:36:42.689429 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:42.689406 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4vsjm_d74970a0-05c1-475e-b4bd-dc5a9e259f21/console-operator/2.log" Jul 20 03:36:43.201660 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:43.201629 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78f8b58b84-vmqph_3440a165-5182-437a-ab26-85e6f7d16654/console/0.log" Jul 20 03:36:43.637799 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:43.637761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" event={"ID":"3536c191-9aec-4cc3-9b59-a34bacf23e48","Type":"ContainerStarted","Data":"dbbdc93a73689f295684a8fbdfa4c5e8ea65158b8b7cc186a8d8bb0d70100cdf"} Jul 20 03:36:43.638194 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:43.637829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:43.638194 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:43.637851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" event={"ID":"3536c191-9aec-4cc3-9b59-a34bacf23e48","Type":"ContainerStarted","Data":"e8b0bba4b9fe01978a6af1a4ee5a371c905111f04496e876ae04dcb2ee8a8ab7"} Jul 20 03:36:43.655055 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:43.655004 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" podStartSLOduration=1.654988442 podStartE2EDuration="1.654988442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 03:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:36:43.652662044 +0000 UTC m=+3098.528463754" watchObservedRunningTime="2026-07-20 03:36:43.654988442 +0000 UTC m=+3098.530790152" Jul 20 03:36:44.907609 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:44.907578 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xl7dx_d64b4dc1-c25c-443f-a40a-55c43b1d19f8/dns/0.log" Jul 20 03:36:44.930340 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:44.930310 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xl7dx_d64b4dc1-c25c-443f-a40a-55c43b1d19f8/kube-rbac-proxy/0.log" Jul 20 03:36:44.954828 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:44.954804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gb6bp_b41a9a75-9ff7-4235-a372-66fc6b1dc25e/dns-node-resolver/0.log" Jul 20 03:36:45.511048 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:45.511019 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7564878f47-6bcdn_46026170-a1e1-4f36-97dc-f4c9fc2de40c/registry/0.log" Jul 20 03:36:45.579675 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:45.579651 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pl4t4_ea855527-0972-4f91-8790-fb5b6217bd38/node-ca/0.log" Jul 20 03:36:46.649264 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:46.649233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d88568984-wwmc2_98682004-cb14-4a45-afea-0f41e3695832/kube-auth-proxy/0.log" Jul 20 03:36:46.736949 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:46.736918 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bltbd_65f000f6-0a19-4e6c-aded-34677ef8fc3d/payload-pre-processing/0.log" Jul 20 03:36:46.762468 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:46.762444 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hk2g6_773f6e47-f1d3-4555-8680-c23e0ce263c3/payload-processing/0.log" Jul 20 03:36:47.512089 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:47.512059 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnzc7_993b4933-d42d-4203-b148-ca51bfb26d38/serve-healthcheck-canary/0.log" Jul 20 03:36:47.983514 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:47.983484 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwggl_122e6085-1ccb-4bd9-bdcf-dd77366c6652/kube-rbac-proxy/0.log" Jul 20 03:36:48.002194 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:48.002166 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwggl_122e6085-1ccb-4bd9-bdcf-dd77366c6652/exporter/0.log" Jul 20 03:36:48.021469 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:48.021442 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwggl_122e6085-1ccb-4bd9-bdcf-dd77366c6652/extractor/0.log" Jul 20 03:36:49.653016 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:49.652989 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-4vdcd" Jul 20 03:36:50.227875 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:50.227846 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pb9bv_1b6c66e2-5522-4894-a420-3e0381a84cbf/manager/0.log" Jul 20 03:36:50.395843 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:50.395811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5msd8_abfb8c3c-fd68-4a72-ae16-5f30b8e09d2d/manager/0.log" Jul 20 03:36:50.501079 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:50.500992 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645d8c8bc6-hxh5g_fda65796-3ea2-47ec-8b5f-b6db2b173d31/manager/0.log" Jul 20 03:36:50.612940 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:50.612908 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb_11461e94-3b0d-4e4e-9137-995511b4ceb0/manager/0.log" Jul 20 03:36:50.632138 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:50.632115 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-7zjnb_11461e94-3b0d-4e4e-9137-995511b4ceb0/copy-manifests/0.log" Jul 20 03:36:52.035568 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:52.032045 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vp4q7_50b8853a-131b-41f6-959a-660ad5b6812a/openshift-lws-operator/0.log" Jul 20 03:36:52.570644 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:52.570561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d88568984-wwmc2_98682004-cb14-4a45-afea-0f41e3695832/kube-auth-proxy/0.log" Jul 20 03:36:52.647770 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:52.647729 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bltbd_65f000f6-0a19-4e6c-aded-34677ef8fc3d/payload-pre-processing/0.log" Jul 20 03:36:52.666565 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:52.666522 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hk2g6_773f6e47-f1d3-4555-8680-c23e0ce263c3/payload-processing/0.log" Jul 20 03:36:54.058137 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:54.058102 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d88568984-wwmc2_98682004-cb14-4a45-afea-0f41e3695832/kube-auth-proxy/0.log" Jul 20 03:36:54.137131 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:54.137101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bltbd_65f000f6-0a19-4e6c-aded-34677ef8fc3d/payload-pre-processing/0.log" Jul 20 03:36:54.154211 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:54.154180 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hk2g6_773f6e47-f1d3-4555-8680-c23e0ce263c3/payload-processing/0.log" Jul 20 03:36:58.685262 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.685226 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/kube-multus-additional-cni-plugins/0.log" Jul 20 03:36:58.703282 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.703253 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/egress-router-binary-copy/0.log" Jul 20 03:36:58.721072 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.721048 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/cni-plugins/0.log" Jul 20 03:36:58.740684 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.740659 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/bond-cni-plugin/0.log" Jul 20 03:36:58.758309 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.758278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/routeoverride-cni/0.log" Jul 20 03:36:58.776332 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.776278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/whereabouts-cni-bincopy/0.log" Jul 20 03:36:58.794820 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.794799 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkr6c_dfc4af98-d2b1-4db8-9acd-17e9a8c3b99d/whereabouts-cni/0.log" Jul 20 03:36:58.856083 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.856055 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mp6b8_eaeb2db2-e90d-4c34-884e-41f944356963/kube-multus/0.log" Jul 20 03:36:58.916822 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.916794 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2vxkh_0cbf1d55-411c-48ce-b16f-33b6adf859e7/network-metrics-daemon/0.log" Jul 20 03:36:58.934227 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:58.934208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2vxkh_0cbf1d55-411c-48ce-b16f-33b6adf859e7/kube-rbac-proxy/0.log" Jul 20 03:36:59.935961 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:59.935930 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-controller/0.log" Jul 20 03:36:59.950691 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:59.950673 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/1.log" Jul 20 03:36:59.950811 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:59.950794 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovn-acl-logging/0.log" Jul 20 03:36:59.968177 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:59.968152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/kube-rbac-proxy-node/0.log" Jul 20 03:36:59.988722 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:36:59.988698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/kube-rbac-proxy-ovn-metrics/0.log" Jul 20 03:37:00.005837 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:37:00.005811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/northd/0.log" Jul 20 03:37:00.025138 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:37:00.025114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/nbdb/0.log" Jul 20 03:37:00.044896 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:37:00.044870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/sbdb/0.log" Jul 20 03:37:00.096962 ip-10-0-134-38 kubenswrapper[2565]: I0720 03:37:00.096920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzx5_4fcacf48-1d67-4537-ad9e-64714f2856b1/ovnkube-controller/0.log"