Jul 22 11:23:49.727233 ip-10-0-136-40 systemd[1]: Starting Kubernetes Kubelet... Jul 22 11:23:50.099050 ip-10-0-136-40 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:23:50.099050 ip-10-0-136-40 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 11:23:50.099050 ip-10-0-136-40 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:23:50.099050 ip-10-0-136-40 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 11:23:50.099050 ip-10-0-136-40 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:23:50.101911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.101826 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 11:23:50.107189 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107166 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.107189 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107185 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.107189 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107191 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.107189 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107196 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107296 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107304 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107309 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107313 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107318 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107322 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107326 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107331 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107335 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107339 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107343 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107347 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107351 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107355 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107359 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.107358 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107363 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107367 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107372 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107379 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107383 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107388 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107392 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107396 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107400 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107404 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107408 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107412 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107416 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107421 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107424 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107428 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107432 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107437 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107441 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107445 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.107736 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107448 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107453 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107466 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107470 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107474 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107477 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107479 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107482 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107484 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107489 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107492 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107496 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107499 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107502 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107505 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107507 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107510 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107513 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107515 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107518 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.108241 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107521 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107523 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107526 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107528 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107531 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107533 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107536 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107539 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107541 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107545 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107549 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107552 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107556 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107559 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107562 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107564 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107567 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107570 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107572 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107575 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.108844 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107577 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107580 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107583 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107585 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107588 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107591 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107594 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107597 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107599 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107602 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107604 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107606 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107609 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107611 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107614 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107616 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107618 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107621 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107625 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.109343 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.107628 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108883 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108892 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108896 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108900 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108903 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108905 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108908 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108911 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108914 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108917 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108920 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108922 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108925 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108927 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108930 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108933 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108936 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108939 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108942 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.109831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108945 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108947 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108964 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108967 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108970 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108972 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108975 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108978 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108980 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108983 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108985 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108988 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108991 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108993 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108995 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.108998 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109001 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109003 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109005 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109008 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.110325 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109010 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109014 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109017 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109020 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109023 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109025 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109027 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109031 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109034 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109037 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109040 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109043 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109046 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109049 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109051 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109054 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109056 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109058 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109061 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109064 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.110839 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109066 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109068 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109071 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109073 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109076 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109078 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109080 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109083 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109085 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109088 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109090 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109093 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109095 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109098 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109100 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109103 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109105 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109107 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109110 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109113 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.111335 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109116 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109118 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109121 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109124 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109128 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109130 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109133 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109135 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109139 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109141 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109144 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109147 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109149 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109151 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109154 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109156 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109159 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109161 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109164 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.111816 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109168 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109538 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109543 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109546 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109549 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109553 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109557 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109560 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109563 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109566 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109569 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109572 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109575 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109578 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109581 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109584 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109587 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109590 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109593 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.112279 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109596 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109599 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109602 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109605 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109608 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109610 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109613 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109616 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109618 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109621 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109623 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109626 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109628 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109631 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109633 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109635 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109638 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109640 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109642 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109645 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.112735 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109647 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109649 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109652 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109655 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109657 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109660 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109662 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109665 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109668 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109670 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109673 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109676 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109679 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109681 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109684 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109686 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109688 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109691 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109693 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.113244 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109696 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109698 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109700 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109702 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109705 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109707 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109710 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109712 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109714 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109717 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109719 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109722 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109724 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109727 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109729 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109732 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109734 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109736 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109741 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109744 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.113704 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109746 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109751 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109755 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109758 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109761 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109765 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109769 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109772 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109774 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109777 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109779 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109782 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109784 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109787 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109789 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109792 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109794 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109796 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109799 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.114192 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109801 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109803 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.109806 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109883 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109893 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109903 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109909 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109914 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109919 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109925 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109931 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109935 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109938 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109941 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109945 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109949 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109964 2569 flags.go:64] FLAG: --cgroup-root="" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109967 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109970 2569 flags.go:64] FLAG: --client-ca-file="" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109973 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109976 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109982 2569 flags.go:64] FLAG: --cluster-domain="" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109984 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109987 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 11:23:50.114650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109990 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.109994 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110002 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110005 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110008 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110011 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110014 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110017 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110020 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110023 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110026 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110030 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110033 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110035 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110038 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110041 2569 flags.go:64] FLAG: --enable-server="true" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110044 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110048 2569 flags.go:64] FLAG: --event-burst="100" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110051 2569 flags.go:64] FLAG: --event-qps="50" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110054 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110057 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110059 2569 flags.go:64] FLAG: --eviction-hard="" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110065 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110068 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110071 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 11:23:50.115240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110074 2569 flags.go:64] FLAG: --eviction-soft="" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110077 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110080 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110083 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110086 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110089 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110091 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110094 2569 flags.go:64] FLAG: --feature-gates="" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110098 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110101 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110104 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110107 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110110 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110113 2569 flags.go:64] FLAG: --help="false" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110116 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110118 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110121 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110124 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110127 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110131 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110133 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110136 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110139 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 11:23:50.115859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110141 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110144 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110147 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110150 2569 flags.go:64] FLAG: --kube-reserved="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110152 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110155 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110159 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110162 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110164 2569 flags.go:64] FLAG: --lock-file="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110167 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110170 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110173 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110178 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110181 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110184 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110187 2569 flags.go:64] FLAG: --logging-format="text" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110190 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110193 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110195 2569 flags.go:64] FLAG: --manifest-url="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110198 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110202 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110205 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110209 2569 flags.go:64] FLAG: --max-pods="110" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110211 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110214 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 11:23:50.116454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110217 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110220 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110223 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110228 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110232 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110243 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110246 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110249 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110252 2569 flags.go:64] FLAG: --pod-cidr="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110255 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110261 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110264 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110266 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110270 2569 flags.go:64] FLAG: --port="10250" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110273 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110277 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03433342c10abe016" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110280 2569 flags.go:64] FLAG: --qos-reserved="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110283 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110286 2569 flags.go:64] FLAG: --register-node="true" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110288 2569 flags.go:64] FLAG: --register-with-taints="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110292 2569 flags.go:64] FLAG: --registry-burst="10" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110295 2569 flags.go:64] FLAG: --registry-qps="5" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110298 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110301 2569 flags.go:64] FLAG: --reserved-memory="" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110304 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 11:23:50.117155 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110307 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110310 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110313 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110315 2569 flags.go:64] FLAG: --runonce="false" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110318 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110321 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110323 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110326 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110329 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110332 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110335 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110338 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110341 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110343 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110349 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110351 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110354 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110357 2569 flags.go:64] FLAG: --system-cgroups="" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110360 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110365 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110368 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110372 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110376 2569 flags.go:64] FLAG: --tls-min-version="" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110378 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110381 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 11:23:50.118210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110384 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110387 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110390 2569 flags.go:64] FLAG: --v="2" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110393 2569 flags.go:64] FLAG: --version="false" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110399 2569 flags.go:64] FLAG: --vmodule="" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110403 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.110406 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110524 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110528 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110532 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110535 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110538 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110541 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110544 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110546 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110549 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110552 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110555 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110557 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110560 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110562 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.119254 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110566 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110569 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110571 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110574 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110576 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110579 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110583 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110585 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110587 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110590 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110592 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110594 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110597 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110600 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110602 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110604 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110607 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110610 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110612 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110615 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.119781 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110617 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110619 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110622 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110624 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110626 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110629 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110631 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110634 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110636 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110639 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110641 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110643 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110649 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110651 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110654 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110656 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110658 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110661 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110664 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110667 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.120505 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110669 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110672 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110675 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110677 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110679 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110682 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110684 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110687 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110689 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110692 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110695 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110697 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110701 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110705 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110708 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110710 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110713 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110715 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110718 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110720 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.121401 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110723 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110725 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110728 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110730 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110734 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110736 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110739 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110741 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110744 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110747 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110750 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110753 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110756 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110760 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110763 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110765 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110768 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110771 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110774 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.122033 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110776 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110779 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110781 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110793 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110795 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.110798 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.111469 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.119225 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.119244 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.120086 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.120104 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120191 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120200 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120205 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120209 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.122681 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120214 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120219 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120224 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120229 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120233 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120237 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120242 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120246 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120250 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120254 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120258 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120263 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120267 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120271 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120275 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120279 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120284 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120288 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120292 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120296 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.123345 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120300 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120306 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120310 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120315 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120319 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120324 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120329 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120333 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120338 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120342 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120346 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120351 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120355 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120359 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120364 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120368 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120372 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120376 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120380 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120387 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.124000 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120393 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120400 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120406 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120412 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120418 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120422 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120427 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120432 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120436 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120441 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120446 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120450 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120455 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120459 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120465 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120471 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120475 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120481 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120486 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.124595 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120490 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120494 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120498 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120502 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120506 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120510 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120514 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120518 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120521 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120527 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120533 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120538 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120543 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120547 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120552 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120557 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120563 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120568 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120573 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.125126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120577 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120582 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120586 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120591 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120596 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120600 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120604 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120609 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120614 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120618 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120622 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120627 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120631 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120636 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120641 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120645 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.125630 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120650 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.120658 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120875 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120884 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120889 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120895 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120899 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120904 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120908 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120912 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120916 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120920 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120927 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120933 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:23:50.126126 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120938 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120942 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120947 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120971 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120978 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120984 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120988 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120993 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.120998 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121002 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121007 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121011 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121016 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121021 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121026 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121030 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121036 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121041 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121045 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:23:50.126475 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121050 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121054 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121058 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121062 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121066 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121070 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121079 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121083 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121088 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121092 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121096 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121100 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121104 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121108 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121112 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121117 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121121 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121125 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121129 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121133 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:23:50.126928 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121137 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121141 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121145 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121149 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121153 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121157 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121161 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121166 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121170 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121174 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121179 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121183 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121187 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121191 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121195 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121198 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121203 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121207 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121211 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121215 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:23:50.127474 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121219 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121223 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121227 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121231 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121234 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121238 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121242 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121246 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121250 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121255 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121259 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121263 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121267 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121271 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121277 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121282 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121287 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121292 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121296 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:23:50.127974 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121300 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121305 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121309 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121313 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121318 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121322 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121326 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121330 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:50.121334 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.121342 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.121490 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.124421 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.125837 2569 server.go:1019] "Starting client certificate rotation" Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.125936 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:23:50.128564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.125982 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:23:50.129158 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.128910 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:23:50.131598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.131576 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:23:50.140255 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.140237 2569 log.go:25] "Validated CRI v1 runtime API" Jul 22 11:23:50.142477 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.142462 2569 log.go:25] "Validated CRI v1 image API" Jul 22 11:23:50.143175 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.143155 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 11:23:50.145494 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.145476 2569 fs.go:135] Filesystem UUIDs: map[01d1388b-42f5-44e1-ae4e-691adf7ff1e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb4f3b98-e661-4670-818d-f09d7cdf094c:/dev/nvme0n1p3] Jul 22 11:23:50.145558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.145494 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 11:23:50.150583 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.150478 2569 manager.go:216] Machine: {Timestamp:2026-07-22 11:23:50.14911567 +0000 UTC m=+0.310658368 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100240 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261c9134efafd51ead4ac689445377 SystemUUID:ec261c91-34ef-afd5-1ead-4ac689445377 BootID:03f1a464-8698-49c8-a093-f3ca1b466566 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:c6:ae:28:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:c6:ae:28:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:1c:e3:a2:15:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 11:23:50.150583 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.150583 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 11:23:50.150701 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.150691 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 11:23:50.153917 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.153893 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 11:23:50.154076 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.153922 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 11:23:50.154643 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.154633 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 11:23:50.154675 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.154645 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 11:23:50.154675 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.154653 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:23:50.155282 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.155271 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:23:50.156915 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.156905 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 11:23:50.160452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160440 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:23:50.160747 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160736 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 11:23:50.160798 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160777 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 22 11:23:50.160798 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160787 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 11:23:50.160871 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160803 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 11:23:50.160871 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160812 2569 kubelet.go:398] "Adding apiserver pod source" Jul 22 11:23:50.160871 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.160827 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 11:23:50.161165 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.161153 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:23:50.161226 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.161172 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:23:50.162789 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.162761 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:23:50.163110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.163095 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 11:23:50.164721 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.164708 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 11:23:50.164771 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.164739 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 11:23:50.165322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165307 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 11:23:50.165386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165324 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 11:23:50.165386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165334 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 11:23:50.165386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165343 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 11:23:50.165386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165355 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 11:23:50.165386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165363 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165420 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165426 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165433 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165439 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165461 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 11:23:50.165598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.165469 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 11:23:50.166819 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.166809 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 11:23:50.166819 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.166819 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 11:23:50.170614 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.170591 2569 server.go:1268] "Started kubelet" Jul 22 11:23:50.171349 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.171088 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 11:23:50.171349 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.171321 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 11:23:50.171525 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.171422 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 11:23:50.172263 ip-10-0-136-40 systemd[1]: Started Kubernetes Kubelet. Jul 22 11:23:50.172421 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.172388 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 11:23:50.172493 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.172407 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 11:23:50.172640 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.172622 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 11:23:50.173330 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.173256 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 11:23:50.173383 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.173335 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 22 11:23:50.177490 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.177470 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 11:23:50.180113 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.179020 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-40.ec2.internal.18c498ddab0da685 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-40.ec2.internal,UID:ip-10-0-136-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-40.ec2.internal,},FirstTimestamp:2026-07-22 11:23:50.170551941 +0000 UTC m=+0.332094632,LastTimestamp:2026-07-22 11:23:50.170551941 +0000 UTC m=+0.332094632,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-40.ec2.internal,}" Jul 22 11:23:50.182826 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.182809 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 11:23:50.182924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.182827 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 11:23:50.182992 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.182939 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.183585 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183567 2569 factory.go:55] Registering systemd factory Jul 22 11:23:50.183689 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183678 2569 factory.go:223] Registration of the systemd container factory successfully Jul 22 11:23:50.183833 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183650 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 11:23:50.183935 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183836 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 11:23:50.183935 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183647 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 11:23:50.184114 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183945 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 22 11:23:50.184114 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.183971 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 22 11:23:50.184193 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184176 2569 factory.go:153] Registering CRI-O factory Jul 22 11:23:50.184193 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184187 2569 factory.go:223] Registration of the crio container factory successfully Jul 22 11:23:50.184287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184238 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 11:23:50.184287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184255 2569 factory.go:103] Registering Raw factory Jul 22 11:23:50.184287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184265 2569 manager.go:1196] Started watching for new ooms in manager Jul 22 11:23:50.184578 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.184553 2569 manager.go:349] Starting recovery of all containers Jul 22 11:23:50.188615 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.188593 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 11:23:50.188615 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.188597 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 11:23:50.194294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.194157 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nfff" Jul 22 11:23:50.194688 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.194671 2569 manager.go:354] Recovery completed Jul 22 11:23:50.198522 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.198505 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.200280 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200265 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.200364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200300 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.200364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200315 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.200862 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200847 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.200937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200847 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 11:23:50.200937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200905 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 11:23:50.200937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.200924 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:23:50.201537 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.201524 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.201603 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.201547 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.201603 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.201564 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.202189 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.202175 2569 policy_none.go:49] "None policy: Start" Jul 22 11:23:50.202255 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.202194 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 11:23:50.202255 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.202207 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 11:23:50.202941 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.202922 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nfff" Jul 22 11:23:50.203051 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.202937 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-40.ec2.internal.18c498ddacd34743 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-40.ec2.internal,UID:ip-10-0-136-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-40.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-40.ec2.internal,},FirstTimestamp:2026-07-22 11:23:50.200280899 +0000 UTC m=+0.361823590,LastTimestamp:2026-07-22 11:23:50.200280899 +0000 UTC m=+0.361823590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-40.ec2.internal,}" Jul 22 11:23:50.204586 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.204570 2569 policy_none.go:47] "Start" Jul 22 11:23:50.246464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246447 2569 manager.go:337] "Starting Device Plugin manager" Jul 22 11:23:50.246553 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.246478 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 11:23:50.246553 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246488 2569 server.go:87] "Starting device plugin registration server" Jul 22 11:23:50.246755 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246743 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 11:23:50.246791 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246759 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 11:23:50.246875 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246858 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 11:23:50.247010 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.246997 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 11:23:50.247010 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.247010 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 11:23:50.247436 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.247418 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 11:23:50.247504 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.247460 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.342176 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.342141 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 11:23:50.343438 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.343410 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 11:23:50.343488 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.343444 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 11:23:50.343488 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.343468 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 11:23:50.343577 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.343500 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 11:23:50.346036 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.346014 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:50.347473 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.347457 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.348285 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.348271 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.348345 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.348302 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.348345 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.348315 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.348345 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.348336 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.355035 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.354994 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.355035 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.355016 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-40.ec2.internal\": node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.383247 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.383224 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.444392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.444366 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal"] Jul 22 11:23:50.444471 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.444431 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.446065 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.446046 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.446138 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.446079 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.446138 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.446094 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.447434 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.447422 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.447585 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.447572 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.447620 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.447600 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.454208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454184 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.454208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454211 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.454355 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454224 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.454355 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454244 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.454355 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454226 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.454355 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.454259 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.455567 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.455552 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.455630 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.455585 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:23:50.456326 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.456310 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:23:50.456389 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.456340 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:23:50.456389 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.456355 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:23:50.483382 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.483364 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.484651 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.484632 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-40.ec2.internal\" not found" node="ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.489074 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.489058 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-40.ec2.internal\" not found" node="ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.583673 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.583647 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.584830 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.584811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42c8e91a3838fc74d7daaaa54e385ed8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-40.ec2.internal\" (UID: \"42c8e91a3838fc74d7daaaa54e385ed8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.584903 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.584842 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.584903 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.584859 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.684139 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.684057 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.685187 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.685232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.685232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42c8e91a3838fc74d7daaaa54e385ed8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-40.ec2.internal\" (UID: \"42c8e91a3838fc74d7daaaa54e385ed8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.685325 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42c8e91a3838fc74d7daaaa54e385ed8-config\") pod \"kube-apiserver-proxy-ip-10-0-136-40.ec2.internal\" (UID: \"42c8e91a3838fc74d7daaaa54e385ed8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.685325 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685270 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.685325 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.685299 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7c88fdb2851902bde044611542b041-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal\" (UID: \"1d7c88fdb2851902bde044611542b041\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.784489 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.784452 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.786626 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.786612 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.793875 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:50.793852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:50.884552 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.884514 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:50.985514 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:50.985437 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.046196 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.046171 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:51.085796 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:51.085768 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.125627 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.125597 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 11:23:51.126221 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.125755 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 11:23:51.126221 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.125769 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 11:23:51.186324 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:51.186297 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.208576 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.208501 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 11:18:50 +0000 UTC" deadline="2028-03-09 20:48:29.470880492 +0000 UTC" Jul 22 11:23:51.208576 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.208565 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14313h24m38.26231853s" Jul 22 11:23:51.287170 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:51.287141 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.363785 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:51.363745 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7c88fdb2851902bde044611542b041.slice/crio-fe1c47f8d11f9c6502f555333d826003a6ec07a837e70e47ec86ef34c50364b9 WatchSource:0}: Error finding container fe1c47f8d11f9c6502f555333d826003a6ec07a837e70e47ec86ef34c50364b9: Status 404 returned error can't find the container with id fe1c47f8d11f9c6502f555333d826003a6ec07a837e70e47ec86ef34c50364b9 Jul 22 11:23:51.364322 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:51.364287 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c8e91a3838fc74d7daaaa54e385ed8.slice/crio-f3182613a56848e290159917ff142eaca01ce6c92a9bf75650a5b7617e1141c7 WatchSource:0}: Error finding container f3182613a56848e290159917ff142eaca01ce6c92a9bf75650a5b7617e1141c7: Status 404 returned error can't find the container with id f3182613a56848e290159917ff142eaca01ce6c92a9bf75650a5b7617e1141c7 Jul 22 11:23:51.369450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.369431 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:23:51.388309 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:51.388278 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.439958 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.438620 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:51.488415 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:51.488342 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-40.ec2.internal\" not found" Jul 22 11:23:51.584000 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.583955 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:51.683209 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.683167 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" Jul 22 11:23:51.702170 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.702142 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:23:51.703274 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.703258 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" Jul 22 11:23:51.717422 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:51.717399 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:23:52.161628 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.161594 2569 apiserver.go:52] "Watching apiserver" Jul 22 11:23:52.166938 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.166905 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 11:23:52.168970 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.168930 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fq45z","kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rfqm5","openshift-image-registry/node-ca-p6rxv","openshift-multus/multus-bgbzz","openshift-multus/network-metrics-daemon-7qbhz","openshift-network-diagnostics/network-check-target-5wn2g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd","openshift-dns/node-resolver-579l5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal","openshift-multus/multus-additional-cni-plugins-lhp7f","openshift-network-operator/iptables-alerter-rrc7z","openshift-ovn-kubernetes/ovnkube-node-5smx8","kube-system/global-pull-secret-syncer-jwjv7"] Jul 22 11:23:52.172106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.171644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:52.172106 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.171745 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:23:52.174901 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.174876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.177637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.177612 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.178075 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.177965 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4cc9z\"" Jul 22 11:23:52.178616 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.178308 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 11:23:52.178616 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.178493 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 11:23:52.178763 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.178681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 11:23:52.179004 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.178985 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.179242 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.179206 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 11:23:52.184367 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.183841 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.184742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.184716 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 11:23:52.186363 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.186321 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.186363 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.186358 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7qvls\"" Jul 22 11:23:52.187587 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.186730 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.187587 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.186847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.188858 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.188838 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.189149 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.189132 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.189797 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.189449 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5kh7h\"" Jul 22 11:23:52.189797 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.189650 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 11:23:52.190369 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.190350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.192663 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.192638 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.192884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.192863 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 11:23:52.193188 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193158 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nz2gj\"" Jul 22 11:23:52.193283 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.193342 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.193279 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:23:52.193400 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193358 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 11:23:52.193569 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193551 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.193636 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-config\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.193636 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-run\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.193708 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193646 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tw8s\" (UniqueName: \"kubernetes.io/projected/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-kube-api-access-8tw8s\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.193708 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-kubelet\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.193708 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-var-lib-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.193859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-log-socket\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.193859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193746 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-netd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.193859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.193859 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193818 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-systemd-units\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193904 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysconfig\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-systemd\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.193964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289vm\" (UniqueName: \"kubernetes.io/projected/89dc48b3-1d1d-446f-971c-0ae831206a4e-kube-api-access-289vm\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-serviceca\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-ovn\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-env-overrides\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-sys\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-host\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194175 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-etc-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-node-log\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-bin\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-modprobe-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:52.194426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-var-lib-kubelet\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194806 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-netns\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194806 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194795 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-systemd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194913 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-script-lib\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.194913 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-lib-modules\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.194913 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-host\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.195106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.194943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-tuned\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.196172 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196147 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-tmp\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.196284 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-slash\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.196284 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.196507 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.196507 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovn-node-metrics-cert\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.196507 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66t4\" (UniqueName: \"kubernetes.io/projected/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-kube-api-access-z66t4\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.196507 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-kubernetes\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.196507 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.196478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-conf\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.200711 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.200198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.202327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.200903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.202327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.201703 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:23:52.202327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.202282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rc97p\"" Jul 22 11:23:52.202327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.202306 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.202327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.202303 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.202747 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.202731 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.202877 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.202819 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:23:52.203411 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.203390 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 11:23:52.203760 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.203730 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bxlfx\"" Jul 22 11:23:52.203850 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.203780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 11:23:52.205949 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.205930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.208066 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.208024 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 11:23:52.208186 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.208047 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.208328 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.208298 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hbl5s\"" Jul 22 11:23:52.208414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.208379 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.208665 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.208645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.210836 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.210818 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 11:23:52.211219 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.211100 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjwrh\"" Jul 22 11:23:52.211365 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.211345 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 11:23:52.211470 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.211392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.213482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.213055 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:23:52.213482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.213318 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 11:23:52.213482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.213355 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcn57\"" Jul 22 11:23:52.213945 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.213890 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 11:23:52.237432 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.237405 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqmhv" Jul 22 11:23:52.251165 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.251139 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqmhv" Jul 22 11:23:52.285644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.285618 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 11:23:52.297031 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-ovn\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-host\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxr88\" (UniqueName: \"kubernetes.io/projected/d89a345f-130b-45e0-b661-768c66f0fbcb-kube-api-access-lxr88\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-netns\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkvn\" (UniqueName: \"kubernetes.io/projected/52b81a27-4b30-472f-9bc6-e49f59d17793-kube-api-access-xwkvn\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-node-log\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.297181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-dbus\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-var-lib-kubelet\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297253 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-kubelet\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-conf-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-netns\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-host\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-tuned\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-socket-dir-parent\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-daemon-config\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299c4307-97a1-4299-b80f-9cda990540a2-tmp-dir\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gggg\" (UniqueName: \"kubernetes.io/projected/299c4307-97a1-4299-b80f-9cda990540a2-kube-api-access-5gggg\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.297490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovn-node-metrics-cert\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66t4\" (UniqueName: \"kubernetes.io/projected/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-kube-api-access-z66t4\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-conf\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-config\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-run\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297766 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckwrs\" (UniqueName: \"kubernetes.io/projected/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kube-api-access-ckwrs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-system-cni-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297818 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-cnibin\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-bin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297902 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/365d1954-1ba1-4ad4-a394-243e9431a119-agent-certs\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.298106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-log-socket\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297954 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-device-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.297979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-cni-binary-copy\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298006 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298055 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysconfig\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-k8s-cni-cncf-io\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298131 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-node-log\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298152 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298196 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-run\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-multus\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-env-overrides\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298306 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-var-lib-kubelet\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-sys\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298344 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-log-socket\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298378 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-run-netns\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.298881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-socket-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298407 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-host\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-registration-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298439 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-ovn\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298448 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-sys-fs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-host\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298476 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-os-release\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-hostroot\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-etc-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298544 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-bin\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298552 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-conf\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-modprobe-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298661 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysconfig\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298857 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.298952 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-sys\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299011 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-modprobe-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.299766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299075 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-bin\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-iptables-alerter-script\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299182 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-env-overrides\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/299c4307-97a1-4299-b80f-9cda990540a2-hosts-file\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-config\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-systemd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299281 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-run-systemd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-script-lib\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-lib-modules\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-tmp\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d82\" (UniqueName: \"kubernetes.io/projected/872aa808-2004-4168-af63-311a8c67807d-kube-api-access-68d82\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-host-slash\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-slash\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299512 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-lib-modules\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-kubernetes\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.300666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-kubernetes\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-system-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-etc-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-cnibin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-multus-certs\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299698 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-slash\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzm4\" (UniqueName: \"kubernetes.io/projected/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-kube-api-access-4vzm4\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw8s\" (UniqueName: \"kubernetes.io/projected/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-kube-api-access-8tw8s\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-kubelet-config\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-etc-kubernetes\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-kubelet\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-var-lib-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-kubelet\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-netd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299950 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-var-lib-openvswitch\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-os-release\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.301452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.299995 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-host-cni-netd\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-systemd-units\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-systemd\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300079 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-sysctl-d\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300123 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-systemd-units\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-systemd\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289vm\" (UniqueName: \"kubernetes.io/projected/89dc48b3-1d1d-446f-971c-0ae831206a4e-kube-api-access-289vm\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-serviceca\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300220 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/365d1954-1ba1-4ad4-a394-243e9431a119-konnectivity-ca\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovnkube-script-lib\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.302294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.300676 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-serviceca\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.302792 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.302773 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-etc-tuned\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.303306 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.303151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-ovn-node-metrics-cert\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.304614 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.304591 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89dc48b3-1d1d-446f-971c-0ae831206a4e-tmp\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.309248 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.309225 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:23:52.309336 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.309255 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:23:52.309336 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.309269 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:52.309336 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.309328 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:23:52.809308297 +0000 UTC m=+2.970850989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:52.316059 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.314693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289vm\" (UniqueName: \"kubernetes.io/projected/89dc48b3-1d1d-446f-971c-0ae831206a4e-kube-api-access-289vm\") pod \"tuned-rfqm5\" (UID: \"89dc48b3-1d1d-446f-971c-0ae831206a4e\") " pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.316059 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.315635 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw8s\" (UniqueName: \"kubernetes.io/projected/058ab9d2-04b1-4be5-9bfd-3d8c14900d49-kube-api-access-8tw8s\") pod \"node-ca-p6rxv\" (UID: \"058ab9d2-04b1-4be5-9bfd-3d8c14900d49\") " pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.316939 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.316919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66t4\" (UniqueName: \"kubernetes.io/projected/57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb-kube-api-access-z66t4\") pod \"ovnkube-node-5smx8\" (UID: \"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.348528 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.348468 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" event={"ID":"1d7c88fdb2851902bde044611542b041","Type":"ContainerStarted","Data":"fe1c47f8d11f9c6502f555333d826003a6ec07a837e70e47ec86ef34c50364b9"} Jul 22 11:23:52.349690 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.349655 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" event={"ID":"42c8e91a3838fc74d7daaaa54e385ed8","Type":"ContainerStarted","Data":"f3182613a56848e290159917ff142eaca01ce6c92a9bf75650a5b7617e1141c7"} Jul 22 11:23:52.352604 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.352584 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:52.400920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.400850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-os-release\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.400929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/365d1954-1ba1-4ad4-a394-243e9431a119-konnectivity-ca\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.400958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxr88\" (UniqueName: \"kubernetes.io/projected/d89a345f-130b-45e0-b661-768c66f0fbcb-kube-api-access-lxr88\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-netns\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401026 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkvn\" (UniqueName: \"kubernetes.io/projected/52b81a27-4b30-472f-9bc6-e49f59d17793-kube-api-access-xwkvn\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401030 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-os-release\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401116 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-dbus\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401113 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-netns\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-dbus\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-kubelet\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-conf-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-socket-dir-parent\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401392 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.401371 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.401427 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:23:52.901413148 +0000 UTC m=+3.062955860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401498 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/365d1954-1ba1-4ad4-a394-243e9431a119-konnectivity-ca\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-daemon-config\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401543 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299c4307-97a1-4299-b80f-9cda990540a2-tmp-dir\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gggg\" (UniqueName: \"kubernetes.io/projected/299c4307-97a1-4299-b80f-9cda990540a2-kube-api-access-5gggg\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401649 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckwrs\" (UniqueName: \"kubernetes.io/projected/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kube-api-access-ckwrs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-system-cni-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-cnibin\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-bin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/365d1954-1ba1-4ad4-a394-243e9431a119-agent-certs\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-device-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-cni-binary-copy\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401780 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299c4307-97a1-4299-b80f-9cda990540a2-tmp-dir\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401793 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-k8s-cni-cncf-io\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-multus\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401835 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-daemon-config\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-socket-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-registration-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.401884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401881 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-kubelet\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-sys-fs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-os-release\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-conf-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-hostroot\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-iptables-alerter-script\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-socket-dir-parent\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/299c4307-97a1-4299-b80f-9cda990540a2-hosts-file\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.401988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-multus-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d82\" (UniqueName: \"kubernetes.io/projected/872aa808-2004-4168-af63-311a8c67807d-kube-api-access-68d82\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402007 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-system-cni-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-host-slash\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-system-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-cnibin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.402446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-multus-certs\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzm4\" (UniqueName: \"kubernetes.io/projected/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-kube-api-access-4vzm4\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-kubelet-config\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402330 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-etc-kubernetes\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402395 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-etc-kubernetes\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-host-slash\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402445 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402450 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402480 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-system-cni-dir\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402484 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-bin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402512 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-multus-certs\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402519 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9c9f01e-fd6b-438e-a92c-11fddaaae685-kubelet-config\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-registration-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402552 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402613 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-run-k8s-cni-cncf-io\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-device-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-sys-fs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.403234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402740 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-os-release\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402788 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-hostroot\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-host-var-lib-cni-multus\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.402880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-cnibin\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403167 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/299c4307-97a1-4299-b80f-9cda990540a2-hosts-file\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403178 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52b81a27-4b30-472f-9bc6-e49f59d17793-cni-binary-copy\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403175 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-iptables-alerter-script\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52b81a27-4b30-472f-9bc6-e49f59d17793-cnibin\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872aa808-2004-4168-af63-311a8c67807d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.403434 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.403542 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:23:52.903523594 +0000 UTC m=+3.065066272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-socket-dir\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.404120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.403674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872aa808-2004-4168-af63-311a8c67807d-cni-binary-copy\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.405341 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.405315 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/365d1954-1ba1-4ad4-a394-243e9431a119-agent-certs\") pod \"konnectivity-agent-fq45z\" (UID: \"365d1954-1ba1-4ad4-a394-243e9431a119\") " pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.415820 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.415737 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxr88\" (UniqueName: \"kubernetes.io/projected/d89a345f-130b-45e0-b661-768c66f0fbcb-kube-api-access-lxr88\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.417420 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.417091 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gggg\" (UniqueName: \"kubernetes.io/projected/299c4307-97a1-4299-b80f-9cda990540a2-kube-api-access-5gggg\") pod \"node-resolver-579l5\" (UID: \"299c4307-97a1-4299-b80f-9cda990540a2\") " pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.417420 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.417284 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkvn\" (UniqueName: \"kubernetes.io/projected/52b81a27-4b30-472f-9bc6-e49f59d17793-kube-api-access-xwkvn\") pod \"multus-bgbzz\" (UID: \"52b81a27-4b30-472f-9bc6-e49f59d17793\") " pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.418757 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.418733 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckwrs\" (UniqueName: \"kubernetes.io/projected/34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e-kube-api-access-ckwrs\") pod \"aws-ebs-csi-driver-node-nxcxd\" (UID: \"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.419812 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.419778 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d82\" (UniqueName: \"kubernetes.io/projected/872aa808-2004-4168-af63-311a8c67807d-kube-api-access-68d82\") pod \"multus-additional-cni-plugins-lhp7f\" (UID: \"872aa808-2004-4168-af63-311a8c67807d\") " pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.420985 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.420957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzm4\" (UniqueName: \"kubernetes.io/projected/d0f1199f-874e-4ef1-97ca-d4b4888aa2d3-kube-api-access-4vzm4\") pod \"iptables-alerter-rrc7z\" (UID: \"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3\") " pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.490335 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.490295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:23:52.501622 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.501593 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" Jul 22 11:23:52.517600 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.517571 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6rxv" Jul 22 11:23:52.531736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.531710 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgbzz" Jul 22 11:23:52.539735 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.539709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-579l5" Jul 22 11:23:52.549693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.549670 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:23:52.561551 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.561528 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" Jul 22 11:23:52.569463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.569442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" Jul 22 11:23:52.578287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.578271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrc7z" Jul 22 11:23:52.906298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.906262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:52.906464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.906329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:52.906464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:52.906359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906454 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906479 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906484 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906517 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906531 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906537 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:23:53.906515227 +0000 UTC m=+4.068058124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906556 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:23:53.906549201 +0000 UTC m=+4.068091877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:52.906582 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:52.906579 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:23:53.906562527 +0000 UTC m=+4.068105223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:53.255015 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.254938 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:18:52 +0000 UTC" deadline="2028-04-26 09:13:46.363014832 +0000 UTC" Jul 22 11:23:53.255015 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.255009 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15453h49m53.108009858s" Jul 22 11:23:53.484954 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.484924 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f1199f_874e_4ef1_97ca_d4b4888aa2d3.slice/crio-c533aaf0d9fbcf4b06dc83a517952d4e42b4a0ac114207c81bb6b5a38b36a64f WatchSource:0}: Error finding container c533aaf0d9fbcf4b06dc83a517952d4e42b4a0ac114207c81bb6b5a38b36a64f: Status 404 returned error can't find the container with id c533aaf0d9fbcf4b06dc83a517952d4e42b4a0ac114207c81bb6b5a38b36a64f Jul 22 11:23:53.486327 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.486305 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dc48b3_1d1d_446f_971c_0ae831206a4e.slice/crio-c2f524033c8b536c7df852ccdf657c9dd3cfe39cec7689bbed2f75f321e9872e WatchSource:0}: Error finding container c2f524033c8b536c7df852ccdf657c9dd3cfe39cec7689bbed2f75f321e9872e: Status 404 returned error can't find the container with id c2f524033c8b536c7df852ccdf657c9dd3cfe39cec7689bbed2f75f321e9872e Jul 22 11:23:53.508152 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.508125 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b81a27_4b30_472f_9bc6_e49f59d17793.slice/crio-724918c0573ac578fd53709aad0b270a72f9bc9e905080f4965f8e9804974012 WatchSource:0}: Error finding container 724918c0573ac578fd53709aad0b270a72f9bc9e905080f4965f8e9804974012: Status 404 returned error can't find the container with id 724918c0573ac578fd53709aad0b270a72f9bc9e905080f4965f8e9804974012 Jul 22 11:23:53.517646 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.517623 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058ab9d2_04b1_4be5_9bfd_3d8c14900d49.slice/crio-fcbc93154b168d29715f3ed56166a74a0d61a142fb6ae2e399f8269bd4d34bfe WatchSource:0}: Error finding container fcbc93154b168d29715f3ed56166a74a0d61a142fb6ae2e399f8269bd4d34bfe: Status 404 returned error can't find the container with id fcbc93154b168d29715f3ed56166a74a0d61a142fb6ae2e399f8269bd4d34bfe Jul 22 11:23:53.518637 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.518528 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d4d2d6_716c_4c9d_82cb_4aeff9a31dbb.slice/crio-ab398484daaa2b90d477403143ef3f81b3552abf1b84c4f0150ffaf6e5c2d113 WatchSource:0}: Error finding container ab398484daaa2b90d477403143ef3f81b3552abf1b84c4f0150ffaf6e5c2d113: Status 404 returned error can't find the container with id ab398484daaa2b90d477403143ef3f81b3552abf1b84c4f0150ffaf6e5c2d113 Jul 22 11:23:53.520568 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.520543 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365d1954_1ba1_4ad4_a394_243e9431a119.slice/crio-a70d78a9fe05d64915d3b3b21f688a60b67cf5a1f78550df67d05b4e2a8e589e WatchSource:0}: Error finding container a70d78a9fe05d64915d3b3b21f688a60b67cf5a1f78550df67d05b4e2a8e589e: Status 404 returned error can't find the container with id a70d78a9fe05d64915d3b3b21f688a60b67cf5a1f78550df67d05b4e2a8e589e Jul 22 11:23:53.521776 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.521751 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872aa808_2004_4168_af63_311a8c67807d.slice/crio-99b82a3df5b801808f6b6487ed043c803034f7b49727d7cbd00e3716ff569663 WatchSource:0}: Error finding container 99b82a3df5b801808f6b6487ed043c803034f7b49727d7cbd00e3716ff569663: Status 404 returned error can't find the container with id 99b82a3df5b801808f6b6487ed043c803034f7b49727d7cbd00e3716ff569663 Jul 22 11:23:53.523273 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.523248 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299c4307_97a1_4299_b80f_9cda990540a2.slice/crio-c95d364de130d73edcbd958d4424e47ffada35ef5ae77c574c0e6a083d0f90ec WatchSource:0}: Error finding container c95d364de130d73edcbd958d4424e47ffada35ef5ae77c574c0e6a083d0f90ec: Status 404 returned error can't find the container with id c95d364de130d73edcbd958d4424e47ffada35ef5ae77c574c0e6a083d0f90ec Jul 22 11:23:53.523783 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:23:53.523645 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bd0cb0_2803_4b9d_b5fc_4c215cc9cf7e.slice/crio-8a54c12a28a193e1772dcedf37b83d211fd0e22c5c546d198612ac962d50402e WatchSource:0}: Error finding container 8a54c12a28a193e1772dcedf37b83d211fd0e22c5c546d198612ac962d50402e: Status 404 returned error can't find the container with id 8a54c12a28a193e1772dcedf37b83d211fd0e22c5c546d198612ac962d50402e Jul 22 11:23:53.913843 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.913619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:53.914030 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.913892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:53.914030 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.913921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:53.914030 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.913931 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:53.914030 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914032 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:23:55.914010152 +0000 UTC m=+6.075552828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:53.914267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914063 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:53.914267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914105 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:23:55.914093107 +0000 UTC m=+6.075635783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:53.914267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914200 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:23:53.914267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914218 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:23:53.914267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914232 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:53.914457 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:53.914273 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:23:55.914259342 +0000 UTC m=+6.075802053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:53.981839 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:53.981751 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:23:54.255342 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.255241 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:18:52 +0000 UTC" deadline="2028-02-05 07:57:59.410769867 +0000 UTC" Jul 22 11:23:54.255342 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.255286 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13508h34m5.155489456s" Jul 22 11:23:54.346444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.346398 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:54.346623 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:54.346526 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:23:54.347007 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.346968 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:54.347125 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:54.347097 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:23:54.347193 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.347178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:54.347282 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:54.347261 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:23:54.367549 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.366446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" event={"ID":"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e","Type":"ContainerStarted","Data":"8a54c12a28a193e1772dcedf37b83d211fd0e22c5c546d198612ac962d50402e"} Jul 22 11:23:54.371604 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.370581 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-579l5" event={"ID":"299c4307-97a1-4299-b80f-9cda990540a2","Type":"ContainerStarted","Data":"c95d364de130d73edcbd958d4424e47ffada35ef5ae77c574c0e6a083d0f90ec"} Jul 22 11:23:54.377292 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.377193 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fq45z" event={"ID":"365d1954-1ba1-4ad4-a394-243e9431a119","Type":"ContainerStarted","Data":"a70d78a9fe05d64915d3b3b21f688a60b67cf5a1f78550df67d05b4e2a8e589e"} Jul 22 11:23:54.380625 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.380547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" event={"ID":"89dc48b3-1d1d-446f-971c-0ae831206a4e","Type":"ContainerStarted","Data":"c2f524033c8b536c7df852ccdf657c9dd3cfe39cec7689bbed2f75f321e9872e"} Jul 22 11:23:54.383399 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.383319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrc7z" event={"ID":"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3","Type":"ContainerStarted","Data":"c533aaf0d9fbcf4b06dc83a517952d4e42b4a0ac114207c81bb6b5a38b36a64f"} Jul 22 11:23:54.388546 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.387674 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d7c88fdb2851902bde044611542b041" containerID="de5bdb825b93bcd196ef9455466a511bb9f8dd79a51708b3be59a0c3430c059c" exitCode=0 Jul 22 11:23:54.388546 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.387751 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" event={"ID":"1d7c88fdb2851902bde044611542b041","Type":"ContainerDied","Data":"de5bdb825b93bcd196ef9455466a511bb9f8dd79a51708b3be59a0c3430c059c"} Jul 22 11:23:54.396541 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.396427 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" event={"ID":"42c8e91a3838fc74d7daaaa54e385ed8","Type":"ContainerStarted","Data":"82898b0284db24e25e4679bcf541e1ecb015681427db96292b32be75954d5a3d"} Jul 22 11:23:54.437948 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.437829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerStarted","Data":"99b82a3df5b801808f6b6487ed043c803034f7b49727d7cbd00e3716ff569663"} Jul 22 11:23:54.439001 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.438911 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-40.ec2.internal" podStartSLOduration=2.438893064 podStartE2EDuration="2.438893064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:23:54.438861894 +0000 UTC m=+4.600404597" watchObservedRunningTime="2026-07-22 11:23:54.438893064 +0000 UTC m=+4.600435766" Jul 22 11:23:54.454224 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.454126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"ab398484daaa2b90d477403143ef3f81b3552abf1b84c4f0150ffaf6e5c2d113"} Jul 22 11:23:54.467250 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.467221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6rxv" event={"ID":"058ab9d2-04b1-4be5-9bfd-3d8c14900d49","Type":"ContainerStarted","Data":"fcbc93154b168d29715f3ed56166a74a0d61a142fb6ae2e399f8269bd4d34bfe"} Jul 22 11:23:54.480717 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:54.480681 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgbzz" event={"ID":"52b81a27-4b30-472f-9bc6-e49f59d17793","Type":"ContainerStarted","Data":"724918c0573ac578fd53709aad0b270a72f9bc9e905080f4965f8e9804974012"} Jul 22 11:23:55.508001 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:55.507949 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" event={"ID":"1d7c88fdb2851902bde044611542b041","Type":"ContainerStarted","Data":"3401428c98b434d93c312efff275ec1e235b829a4abadf04da0457ebe5360e2c"} Jul 22 11:23:55.528705 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:55.527993 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-40.ec2.internal" podStartSLOduration=3.527968376 podStartE2EDuration="3.527968376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:23:55.527414593 +0000 UTC m=+5.688957297" watchObservedRunningTime="2026-07-22 11:23:55.527968376 +0000 UTC m=+5.689511083" Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:55.939156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:55.939323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:55.939361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.939495 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.939560 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:23:59.939540506 +0000 UTC m=+10.101083185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.939986 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.940060 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:23:59.940023513 +0000 UTC m=+10.101566195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.940143 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.940158 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.940171 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:55.940265 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:55.940209 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:23:59.940196481 +0000 UTC m=+10.101739160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:56.345109 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:56.344270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:56.345109 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:56.344411 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:23:56.345109 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:56.344923 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:56.345109 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:56.345058 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:23:56.345704 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:56.345547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:56.345704 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:56.345662 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:58.356151 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:58.356343 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:58.356826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:58.356953 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:58.357066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:58.357263 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:58.357206 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:23:59.976137 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:59.976104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:23:59.976589 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:59.976151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:23:59.976589 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:23:59.976196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:23:59.976589 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976308 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:59.976589 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976364 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:07.976345519 +0000 UTC m=+18.137888198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:23:59.976765 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976734 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:23:59.976765 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976756 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:23:59.976865 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976768 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:59.976865 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976808 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:24:07.976794639 +0000 UTC m=+18.138337320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:23:59.976865 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976864 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:23:59.977033 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:23:59.976895 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:24:07.976884207 +0000 UTC m=+18.138426885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:24:00.344676 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:00.344645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:00.344816 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:00.344772 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:00.344920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:00.344895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:00.345061 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:00.345002 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:00.345135 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:00.345055 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:00.345188 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:00.345171 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:02.346687 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:02.346652 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:02.347108 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:02.346771 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:02.347108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:02.347080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:02.347108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:02.347091 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:02.347251 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:02.347191 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:02.347322 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:02.347290 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:04.343832 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:04.343804 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:04.344232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:04.343850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:04.344232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:04.343863 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:04.344232 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:04.344002 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:04.344232 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:04.344071 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:04.344232 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:04.344202 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:06.344119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:06.344085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:06.344119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:06.344105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:06.344529 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:06.344191 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:06.344529 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:06.344209 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:06.344529 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:06.344270 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:06.344529 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:06.344355 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:08.040685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.040644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:08.040685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.040687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.040730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040824 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040831 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040846 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040858 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040892 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:24.040874647 +0000 UTC m=+34.202417335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040893 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040915 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:24:24.040907762 +0000 UTC m=+34.202450439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:24:08.041267 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.040991 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:24:24.040971313 +0000 UTC m=+34.202513991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:24:08.343934 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.343888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:08.344132 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.344032 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:08.344461 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.344442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:08.344588 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.344546 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:08.344588 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:08.344582 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:08.344756 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:08.344644 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:10.345111 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:10.345079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:10.345558 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:10.345173 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:10.345558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:10.345273 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:10.345558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:10.345372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:10.345558 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:10.345437 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:10.345558 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:10.345475 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:12.344069 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:12.344037 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:12.344069 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:12.344059 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:12.344584 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:12.344042 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:12.344584 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:12.344150 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:12.344584 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:12.344260 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:12.344584 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:12.344343 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:13.542182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.541867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" event={"ID":"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e","Type":"ContainerStarted","Data":"265f64c9bb80d407317395e8c1a4977aabb8fa1b7811a31c0fbcee4f8330597b"} Jul 22 11:24:13.543427 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.543400 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-579l5" event={"ID":"299c4307-97a1-4299-b80f-9cda990540a2","Type":"ContainerStarted","Data":"c30c48e07b27c2b78212ba0107d85846ad306b583eb404b7d7037ab47eeb0bc4"} Jul 22 11:24:13.545805 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.545379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fq45z" event={"ID":"365d1954-1ba1-4ad4-a394-243e9431a119","Type":"ContainerStarted","Data":"483806e5d2cef50c7ffea5f6f900e42e8cbdd40aa1438134b605dcece65dd39d"} Jul 22 11:24:13.547917 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.547891 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" event={"ID":"89dc48b3-1d1d-446f-971c-0ae831206a4e","Type":"ContainerStarted","Data":"480734b0fee97f11ef22d35e439d0ffddf9fb5aace4a9de4c747e39a5e98c409"} Jul 22 11:24:13.550280 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.550258 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="9cd764acf7ddc5b605cfadf3140e114866c84acdff00134210e9d670b3ae856f" exitCode=0 Jul 22 11:24:13.550379 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.550324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"9cd764acf7ddc5b605cfadf3140e114866c84acdff00134210e9d670b3ae856f"} Jul 22 11:24:13.551942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.551918 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"0c03138d7d574ee8fff5a8ac9385e13c1fd54336cc57526ae7ffae87edb143b0"} Jul 22 11:24:13.553523 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.553491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6rxv" event={"ID":"058ab9d2-04b1-4be5-9bfd-3d8c14900d49","Type":"ContainerStarted","Data":"31efe2481ff581e83f5fb24c0f03dafbdeb886add5b019622bd3e3bd6c799511"} Jul 22 11:24:13.554984 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.554947 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgbzz" event={"ID":"52b81a27-4b30-472f-9bc6-e49f59d17793","Type":"ContainerStarted","Data":"509472f643d5b6ceb09687841eba4c3882ae5556c28baf58835af19b4ec54d98"} Jul 22 11:24:13.561683 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.561645 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-579l5" podStartSLOduration=2.856735712 podStartE2EDuration="22.561633397s" totalImagesPullingTime="19.704897685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:13.561267648 +0000 UTC m=+23.722810343" watchObservedRunningTime="2026-07-22 11:24:13.561633397 +0000 UTC m=+23.723176092" Jul 22 11:24:13.599689 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.599649 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fq45z" podStartSLOduration=2.89231345 podStartE2EDuration="22.599630882s" totalImagesPullingTime="19.707317432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:13.599409336 +0000 UTC m=+23.760952033" watchObservedRunningTime="2026-07-22 11:24:13.599630882 +0000 UTC m=+23.761173582" Jul 22 11:24:13.599856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.599833 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bgbzz" podStartSLOduration=2.843014149 podStartE2EDuration="22.599828573s" totalImagesPullingTime="19.756814424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:13.582256143 +0000 UTC m=+23.743798863" watchObservedRunningTime="2026-07-22 11:24:13.599828573 +0000 UTC m=+23.761371303" Jul 22 11:24:13.622558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.622517 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rfqm5" podStartSLOduration=2.898472499 podStartE2EDuration="22.622501391s" totalImagesPullingTime="19.724028892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:13.622417362 +0000 UTC m=+23.783960058" watchObservedRunningTime="2026-07-22 11:24:13.622501391 +0000 UTC m=+23.784044065" Jul 22 11:24:13.642364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.642332 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6rxv" podStartSLOduration=2.928494064 podStartE2EDuration="22.642319235s" totalImagesPullingTime="19.713825171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:13.64208632 +0000 UTC m=+23.803629017" watchObservedRunningTime="2026-07-22 11:24:13.642319235 +0000 UTC m=+23.803861931" Jul 22 11:24:13.785857 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.785832 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:13.992085 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:13.992003 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:14.344075 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.344039 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:14.344310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.344043 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:14.344310 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:14.344162 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:14.344310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.344178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:14.344433 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:14.344308 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:14.344433 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:14.344397 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:14.560372 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.560329 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrc7z" event={"ID":"d0f1199f-874e-4ef1-97ca-d4b4888aa2d3","Type":"ContainerStarted","Data":"3cdb3ec3e92bcb6d177ec121673715e9eaa8539e406a1299e0cee5d328588918"} Jul 22 11:24:14.563641 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.563598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"7baba0216588e6c44bdf9ef74f233b14ff27ab664a4561611c744d1ba4da327e"} Jul 22 11:24:14.563788 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.563679 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"c1e4ffab30168a3fc703e9b0775d380bc1a18e20bd42bd3704ebd86855f1eb34"} Jul 22 11:24:14.563788 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.563731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"25d4caf9353f6ddab68adf95ed50ed66355184a1d8e0639f3c549999083ae459"} Jul 22 11:24:14.563878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.563790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"edaca9ba275648259cea0b013f678161eeeca1740980a97bbe0e9c1e5a2a1dbd"} Jul 22 11:24:14.563878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.563835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"06169a8082692b29fcaf1aac41a3ae5fb461382a7f359c6964d5002a1cbde5b0"} Jul 22 11:24:14.744855 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.744822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:14.847558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:14.847475 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:15.327201 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:15.327165 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:15.425039 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:15.424995 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:16.344451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.344418 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:16.344920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.344422 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:16.344920 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:16.344548 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:16.344920 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:16.344648 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:16.344920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.344707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:16.344920 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:16.344779 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:16.368017 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.367977 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:16.468862 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.468828 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:16.571739 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:16.571703 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"79e91f2f90a06376d8462c5a856df4346ff64d8d5d02f6691b6908ca8613bf8e"} Jul 22 11:24:18.093354 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.093202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:24:18.093988 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.093933 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:24:18.096056 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.095985 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:24:18.096512 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.096492 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fq45z" Jul 22 11:24:18.111208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.111168 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrc7z" podStartSLOduration=7.378800433 podStartE2EDuration="27.111153457s" totalImagesPullingTime="19.732353024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:14.588045791 +0000 UTC m=+24.749588486" watchObservedRunningTime="2026-07-22 11:24:18.111153457 +0000 UTC m=+28.272696153" Jul 22 11:24:18.169382 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.169359 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 11:24:18.203901 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.203871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:18.266269 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.266162 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T11:24:18.169378121Z","UUID":"415bb609-8490-4afc-be33-5d23559f5dfd","Handler":null,"Name":"","Endpoint":""} Jul 22 11:24:18.267760 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.267728 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 11:24:18.267863 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.267784 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 11:24:18.343924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.343892 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:18.344083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.343894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:18.344083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.344042 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:18.344083 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:18.344039 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:18.344209 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:18.344121 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:18.344253 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:18.344203 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:18.578110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.578076 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="75ca1260d41f427c3930c2b2fe0fbb81f27975933c88bd0d57b6bb3a9c1070bc" exitCode=0 Jul 22 11:24:18.578307 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.578154 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"75ca1260d41f427c3930c2b2fe0fbb81f27975933c88bd0d57b6bb3a9c1070bc"} Jul 22 11:24:18.581898 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.581869 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" event={"ID":"57d4d2d6-716c-4c9d-82cb-4aeff9a31dbb","Type":"ContainerStarted","Data":"4fa603d200c1ee27d98a68df5f1562e7fb361524ddd605c5871356eeac81c7bf"} Jul 22 11:24:18.582026 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.582000 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:18.582253 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.582058 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:18.584399 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.584357 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" event={"ID":"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e","Type":"ContainerStarted","Data":"403a8c2c6ad16cb1c30fc264c35fc22f37b0ded834a49b686401d8c0f233063a"} Jul 22 11:24:18.598446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.598376 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:18.598564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:18.598483 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:19.077640 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:19.077607 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:19.588434 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:19.588400 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" event={"ID":"34bd0cb0-2803-4b9d-b5fc-4c215cc9cf7e","Type":"ContainerStarted","Data":"ba553bd28bb171fbbd73bf5213080708a07da69966a6292d32dfc321b4448e4e"} Jul 22 11:24:19.614266 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:19.614225 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxcxd" podStartSLOduration=3.06271101 podStartE2EDuration="28.614210873s" totalImagesPullingTime="25.551499863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:19.613217251 +0000 UTC m=+29.774759957" watchObservedRunningTime="2026-07-22 11:24:19.614210873 +0000 UTC m=+29.775753569" Jul 22 11:24:19.614466 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:19.614448 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" podStartSLOduration=8.800816066 podStartE2EDuration="28.614443609s" totalImagesPullingTime="19.813627543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:18.634573526 +0000 UTC m=+28.796116222" watchObservedRunningTime="2026-07-22 11:24:19.614443609 +0000 UTC m=+29.775986305" Jul 22 11:24:20.225568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.225363 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qbhz"] Jul 22 11:24:20.225770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.225660 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:20.225770 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:20.225747 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:20.226090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.226069 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwjv7"] Jul 22 11:24:20.226179 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.226170 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:20.226283 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:20.226265 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:20.227211 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.227191 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5wn2g"] Jul 22 11:24:20.227285 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.227266 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:20.227342 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:20.227327 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:20.378219 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.378190 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:20.592700 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.592664 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="8e24792e7fae2b201621802a31eae00909df08de87a4cc08d8114b192c70744b" exitCode=0 Jul 22 11:24:20.593210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:20.592754 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"8e24792e7fae2b201621802a31eae00909df08de87a4cc08d8114b192c70744b"} Jul 22 11:24:22.344053 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:22.344021 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:22.344433 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:22.344028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:22.344433 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:22.344118 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:22.344433 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:22.344195 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:22.344433 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:22.344246 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:22.344433 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:22.344294 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:22.600328 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:22.600297 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="f1fd43368a79e9b99c4e5a8bca8bad195a7908157e9a4dbbc5f74afdb3a7057e" exitCode=0 Jul 22 11:24:22.600483 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:22.600348 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"f1fd43368a79e9b99c4e5a8bca8bad195a7908157e9a4dbbc5f74afdb3a7057e"} Jul 22 11:24:24.070887 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.070711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:24.070887 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.070891 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.070925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.070866 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.070985 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.070996 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4qn26 for pod openshift-network-diagnostics/network-check-target-5wn2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.071038 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.071053 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.071065 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26 podName:d1d0d74c-4ddf-42e8-84df-82156efc1f2e nodeName:}" failed. No retries permitted until 2026-07-22 11:24:56.071034918 +0000 UTC m=+66.232577592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qn26" (UniqueName: "kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26") pod "network-check-target-5wn2g" (UID: "d1d0d74c-4ddf-42e8-84df-82156efc1f2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.071085 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret podName:d9c9f01e-fd6b-438e-a92c-11fddaaae685 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:56.071072281 +0000 UTC m=+66.232614955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret") pod "global-pull-secret-syncer-jwjv7" (UID: "d9c9f01e-fd6b-438e-a92c-11fddaaae685") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:24:24.071331 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.071102 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:24:56.071093361 +0000 UTC m=+66.232636039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:24:24.344780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.344701 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:24.344780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.344714 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:24.345022 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.344816 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5wn2g" podUID="d1d0d74c-4ddf-42e8-84df-82156efc1f2e" Jul 22 11:24:24.345022 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.344857 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qbhz" podUID="d89a345f-130b-45e0-b661-768c66f0fbcb" Jul 22 11:24:24.345022 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:24.344908 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:24.345022 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:24.344998 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwjv7" podUID="d9c9f01e-fd6b-438e-a92c-11fddaaae685" Jul 22 11:24:26.165075 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.165038 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-40.ec2.internal" event="NodeReady" Jul 22 11:24:26.165485 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.165203 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 11:24:26.218411 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.218351 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-psq2l"] Jul 22 11:24:26.220420 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.220385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.221560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.220795 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-785x8"] Jul 22 11:24:26.222682 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.222661 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.222835 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.222816 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 11:24:26.222910 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.222850 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqh28\"" Jul 22 11:24:26.223322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.223157 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 11:24:26.225049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.224781 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 11:24:26.225049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.224819 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 11:24:26.225049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.224894 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dccn9\"" Jul 22 11:24:26.225262 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.225170 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 11:24:26.232987 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.232965 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-psq2l"] Jul 22 11:24:26.234746 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.234697 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-785x8"] Jul 22 11:24:26.344509 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.344480 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:26.344509 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.344501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:26.344763 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.344580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:26.347658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347466 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:24:26.347658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347481 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7fc2b\"" Jul 22 11:24:26.347658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:24:26.347911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347681 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:24:26.347911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs9j5\"" Jul 22 11:24:26.347911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.347857 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:24:26.386427 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0150094a-8a0e-4152-a85f-481cdede5afa-tmp-dir\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.386602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-cert\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.386602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.386602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0150094a-8a0e-4152-a85f-481cdede5afa-config-volume\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.386602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hzt\" (UniqueName: \"kubernetes.io/projected/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-kube-api-access-v6hzt\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.386789 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.386644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czk6z\" (UniqueName: \"kubernetes.io/projected/0150094a-8a0e-4152-a85f-481cdede5afa-kube-api-access-czk6z\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.487685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0150094a-8a0e-4152-a85f-481cdede5afa-tmp-dir\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.487685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-cert\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.487685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.487902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0150094a-8a0e-4152-a85f-481cdede5afa-config-volume\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.487902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hzt\" (UniqueName: \"kubernetes.io/projected/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-kube-api-access-v6hzt\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.487902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czk6z\" (UniqueName: \"kubernetes.io/projected/0150094a-8a0e-4152-a85f-481cdede5afa-kube-api-access-czk6z\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.487902 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:26.487824 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:26.487902 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:26.487886 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:26.987866799 +0000 UTC m=+37.149409493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:26.488111 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.487998 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0150094a-8a0e-4152-a85f-481cdede5afa-tmp-dir\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.488415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.488398 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0150094a-8a0e-4152-a85f-481cdede5afa-config-volume\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.492836 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.492814 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-cert\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.500232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.500180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hzt\" (UniqueName: \"kubernetes.io/projected/92b43cfb-ec0f-4f04-b4df-eb01a3105f06-kube-api-access-v6hzt\") pod \"ingress-canary-785x8\" (UID: \"92b43cfb-ec0f-4f04-b4df-eb01a3105f06\") " pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.504078 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.504051 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czk6z\" (UniqueName: \"kubernetes.io/projected/0150094a-8a0e-4152-a85f-481cdede5afa-kube-api-access-czk6z\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.538685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.538653 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-785x8" Jul 22 11:24:26.691422 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.691383 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-785x8"] Jul 22 11:24:26.697261 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:26.697232 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b43cfb_ec0f_4f04_b4df_eb01a3105f06.slice/crio-ad71f45866b25eac24da9c45da8811014b90cd37cdc0628f38dc2984e5495f40 WatchSource:0}: Error finding container ad71f45866b25eac24da9c45da8811014b90cd37cdc0628f38dc2984e5495f40: Status 404 returned error can't find the container with id ad71f45866b25eac24da9c45da8811014b90cd37cdc0628f38dc2984e5495f40 Jul 22 11:24:26.992811 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:26.992774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:26.993021 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:26.992970 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:26.993089 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:26.993036 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:27.99302029 +0000 UTC m=+38.154562963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:27.564156 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.564119 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-g4rwm"] Jul 22 11:24:27.567087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.566912 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.569145 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.569120 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 11:24:27.569558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.569541 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 11:24:27.570108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.570092 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 11:24:27.570205 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.570131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 11:24:27.570205 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.570131 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q9d8p\"" Jul 22 11:24:27.574411 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.574393 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 11:24:27.580697 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.580673 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-g4rwm"] Jul 22 11:24:27.614344 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.614299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-785x8" event={"ID":"92b43cfb-ec0f-4f04-b4df-eb01a3105f06","Type":"ContainerStarted","Data":"ad71f45866b25eac24da9c45da8811014b90cd37cdc0628f38dc2984e5495f40"} Jul 22 11:24:27.697020 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.696980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-service-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.697192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.697034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.697192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.697122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-tmp\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.697192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.697139 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-snapshots\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.697192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.697156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96174ad0-30ee-46ab-9e18-8eafb79af050-serving-cert\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.697192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.697180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54hc\" (UniqueName: \"kubernetes.io/projected/96174ad0-30ee-46ab-9e18-8eafb79af050-kube-api-access-c54hc\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.732287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.732162 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:27.798042 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-service-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798222 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798222 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-tmp\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798222 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-snapshots\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798375 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96174ad0-30ee-46ab-9e18-8eafb79af050-serving-cert\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798375 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798368 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54hc\" (UniqueName: \"kubernetes.io/projected/96174ad0-30ee-46ab-9e18-8eafb79af050-kube-api-access-c54hc\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798580 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-tmp\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798769 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798739 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-service-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.798942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.798919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/96174ad0-30ee-46ab-9e18-8eafb79af050-snapshots\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.799401 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.799380 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96174ad0-30ee-46ab-9e18-8eafb79af050-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.801758 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.801733 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96174ad0-30ee-46ab-9e18-8eafb79af050-serving-cert\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.808807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.808787 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54hc\" (UniqueName: \"kubernetes.io/projected/96174ad0-30ee-46ab-9e18-8eafb79af050-kube-api-access-c54hc\") pod \"insights-operator-555d9d7885-g4rwm\" (UID: \"96174ad0-30ee-46ab-9e18-8eafb79af050\") " pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:27.832770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.832639 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:27.878253 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:27.878226 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" Jul 22 11:24:28.000402 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:28.000351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:28.000615 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:28.000541 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:28.000685 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:28.000626 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:30.00059975 +0000 UTC m=+40.162142450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:28.494501 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:28.494475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-g4rwm"] Jul 22 11:24:28.621196 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:28.621167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerStarted","Data":"d92457461944ef7211610ab794bf4717febd6a689d96e0ff3243ffecea9a371c"} Jul 22 11:24:28.622637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:28.622609 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" event={"ID":"96174ad0-30ee-46ab-9e18-8eafb79af050","Type":"ContainerStarted","Data":"2cca2148827d287c5732d1fc31d5fe54fbe20e17d459b9bc510327b79b3aefd8"} Jul 22 11:24:29.026226 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.026142 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:29.126631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.126468 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:29.629096 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.629065 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="d92457461944ef7211610ab794bf4717febd6a689d96e0ff3243ffecea9a371c" exitCode=0 Jul 22 11:24:29.629096 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.629102 2569 generic.go:356] "Generic (PLEG): container finished" podID="872aa808-2004-4168-af63-311a8c67807d" containerID="756026cb6c25a9d366c0b314152ca8c40acb506b3efa223f9b3b24d8a2b14de3" exitCode=0 Jul 22 11:24:29.629608 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.629150 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"d92457461944ef7211610ab794bf4717febd6a689d96e0ff3243ffecea9a371c"} Jul 22 11:24:29.629608 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:29.629243 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerDied","Data":"756026cb6c25a9d366c0b314152ca8c40acb506b3efa223f9b3b24d8a2b14de3"} Jul 22 11:24:30.022987 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:30.022887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:30.023146 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:30.023061 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:30.023146 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:30.023125 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:34.02310745 +0000 UTC m=+44.184650126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:30.443148 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:30.442887 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:30.516189 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:30.516155 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:31.410117 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.410080 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:31.531399 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.531317 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:31.636813 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.636778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" event={"ID":"872aa808-2004-4168-af63-311a8c67807d","Type":"ContainerStarted","Data":"ed45d96b0c7bb4a06c8e064dd476ad37cf5e719511ed7b06105d56cd70f60b32"} Jul 22 11:24:31.638284 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.638255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-785x8" event={"ID":"92b43cfb-ec0f-4f04-b4df-eb01a3105f06","Type":"ContainerStarted","Data":"a885e0290a36f3f019fa4eddf0c7f5dd9361d69c5bcaa403ee02f54d41820d66"} Jul 22 11:24:31.639649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.639627 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" event={"ID":"96174ad0-30ee-46ab-9e18-8eafb79af050","Type":"ContainerStarted","Data":"1232fcb1973f11b55653211386d2ac4241f6222b92a8eccd0c03d2138913eaa4"} Jul 22 11:24:31.669972 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.669907 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lhp7f" podStartSLOduration=7.948031046 podStartE2EDuration="40.669892805s" totalImagesPullingTime="32.721861759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:31.666150142 +0000 UTC m=+41.827692840" watchObservedRunningTime="2026-07-22 11:24:31.669892805 +0000 UTC m=+41.831435500" Jul 22 11:24:31.690202 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.690163 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-785x8" podStartSLOduration=1.172387355 podStartE2EDuration="5.690149327s" totalImagesPullingTime="4.517761972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:31.689829033 +0000 UTC m=+41.851371743" watchObservedRunningTime="2026-07-22 11:24:31.690149327 +0000 UTC m=+41.851692023" Jul 22 11:24:31.707690 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.707645 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" podStartSLOduration=1.9920242940000001 podStartE2EDuration="4.707632698s" totalImagesPullingTime="2.715608404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:31.706384404 +0000 UTC m=+41.867927100" watchObservedRunningTime="2026-07-22 11:24:31.707632698 +0000 UTC m=+41.869175395" Jul 22 11:24:31.850701 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:31.850668 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:32.020423 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.020399 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:32.378549 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.378520 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:32.478170 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.478140 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:32.905505 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.905473 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m5bw4"] Jul 22 11:24:32.934624 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.934579 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:32.941158 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.941133 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5bw4"] Jul 22 11:24:32.941274 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.941260 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:32.943372 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.943354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 11:24:32.943689 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.943674 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pxx7\"" Jul 22 11:24:32.943742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.943674 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 11:24:32.944025 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.944009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:32.944061 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.944037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5455f9ae-4a87-468e-8025-adc4622ed566-crio-socket\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:32.944126 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.944114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5455f9ae-4a87-468e-8025-adc4622ed566-data-volume\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:32.944176 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.944134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:32.944176 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:32.944156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fd8\" (UniqueName: \"kubernetes.io/projected/5455f9ae-4a87-468e-8025-adc4622ed566-kube-api-access-m2fd8\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.013446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.013414 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:33.044813 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.044788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5455f9ae-4a87-468e-8025-adc4622ed566-data-volume\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.044907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.044815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.044907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.044834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fd8\" (UniqueName: \"kubernetes.io/projected/5455f9ae-4a87-468e-8025-adc4622ed566-kube-api-access-m2fd8\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.044907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.044894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.045077 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.044924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5455f9ae-4a87-468e-8025-adc4622ed566-crio-socket\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.045077 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.044948 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:33.045077 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.045050 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:33.545023308 +0000 UTC m=+43.706565983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:33.045077 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.045055 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:24:33.045232 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.045118 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:33.545101635 +0000 UTC m=+43.706644332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : configmap "kube-rbac-proxy" not found Jul 22 11:24:33.045232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.045156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5455f9ae-4a87-468e-8025-adc4622ed566-data-volume\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.045232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.045198 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5455f9ae-4a87-468e-8025-adc4622ed566-crio-socket\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.057006 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.056976 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fd8\" (UniqueName: \"kubernetes.io/projected/5455f9ae-4a87-468e-8025-adc4622ed566-kube-api-access-m2fd8\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.547431 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.547392 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.547764 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:33.547446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:33.547764 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.547530 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:24:33.547764 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.547542 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:33.547764 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.547592 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:34.547576849 +0000 UTC m=+44.709119523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : configmap "kube-rbac-proxy" not found Jul 22 11:24:33.547764 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:33.547606 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:34.547599122 +0000 UTC m=+44.709141796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:34.049850 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.049817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:34.050027 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.049985 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:34.050072 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.050047 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:42.050029922 +0000 UTC m=+52.211572596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:34.481675 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.481589 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:34.553159 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.553130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:34.553535 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.553180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:34.553535 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.553260 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:24:34.553535 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.553296 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:34.553535 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.553307 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:36.5532949 +0000 UTC m=+46.714837575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : configmap "kube-rbac-proxy" not found Jul 22 11:24:34.553535 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:34.553353 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:36.553336714 +0000 UTC m=+46.714879402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:34.571493 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.571461 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:34.670050 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:34.670030 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-579l5_299c4307-97a1-4299-b80f-9cda990540a2/dns-node-resolver/0.log" Jul 22 11:24:35.468178 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:35.468117 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6rxv_058ab9d2-04b1-4be5-9bfd-3d8c14900d49/node-ca/0.log" Jul 22 11:24:36.268798 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:36.268771 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-785x8_92b43cfb-ec0f-4f04-b4df-eb01a3105f06/serve-healthcheck-canary/0.log" Jul 22 11:24:36.567471 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:36.567440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:36.567648 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:36.567536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:36.567648 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:36.567560 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:24:36.567648 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:36.567618 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:40.567604057 +0000 UTC m=+50.729146733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : configmap "kube-rbac-proxy" not found Jul 22 11:24:36.567801 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:36.567650 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:36.567801 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:36.567723 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:40.567705347 +0000 UTC m=+50.729248035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:37.497071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.497035 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gs8kc"] Jul 22 11:24:37.508648 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.508627 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.511234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.511212 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 11:24:37.511340 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.511214 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 11:24:37.511775 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.511748 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 11:24:37.511882 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.511786 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:24:37.511882 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.511827 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vcrc2\"" Jul 22 11:24:37.515747 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.515685 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gs8kc"] Jul 22 11:24:37.515839 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.515803 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 11:24:37.574883 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.574855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-trusted-ca\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.574883 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.574884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fpb\" (UniqueName: \"kubernetes.io/projected/c40e1c5a-dc8b-4702-a625-6761d7e68330-kube-api-access-z4fpb\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.575069 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.574926 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-config\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.575069 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.575009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c40e1c5a-dc8b-4702-a625-6761d7e68330-serving-cert\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.596901 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.596873 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t"] Jul 22 11:24:37.607941 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.607922 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.609618 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.609596 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t"] Jul 22 11:24:37.610085 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.610057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 11:24:37.610178 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.610128 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:24:37.610290 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.610272 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 11:24:37.610353 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.610319 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q5cv2\"" Jul 22 11:24:37.610353 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.610330 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 11:24:37.676097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b92f2e-6199-4aea-85cc-33afb97f8ea0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.676202 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676124 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b92f2e-6199-4aea-85cc-33afb97f8ea0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.676202 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-config\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.676275 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c40e1c5a-dc8b-4702-a625-6761d7e68330-serving-cert\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.676275 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-trusted-ca\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.676368 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29pp\" (UniqueName: \"kubernetes.io/projected/22b92f2e-6199-4aea-85cc-33afb97f8ea0-kube-api-access-h29pp\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.676368 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fpb\" (UniqueName: \"kubernetes.io/projected/c40e1c5a-dc8b-4702-a625-6761d7e68330-kube-api-access-z4fpb\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.676750 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.676730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-config\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.677106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.677086 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c40e1c5a-dc8b-4702-a625-6761d7e68330-trusted-ca\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.678545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.678530 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c40e1c5a-dc8b-4702-a625-6761d7e68330-serving-cert\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.685723 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.685688 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fpb\" (UniqueName: \"kubernetes.io/projected/c40e1c5a-dc8b-4702-a625-6761d7e68330-kube-api-access-z4fpb\") pod \"console-operator-565b7b569d-gs8kc\" (UID: \"c40e1c5a-dc8b-4702-a625-6761d7e68330\") " pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.706221 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.706199 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh"] Jul 22 11:24:37.718557 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.718538 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p"] Jul 22 11:24:37.718685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.718670 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.720648 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.720627 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 11:24:37.720739 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.720723 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:24:37.721055 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.721035 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 11:24:37.721055 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.721051 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 11:24:37.721187 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.721036 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rctks\"" Jul 22 11:24:37.729489 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.729451 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz"] Jul 22 11:24:37.729580 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.729555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" Jul 22 11:24:37.732273 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.732252 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tpw7x\"" Jul 22 11:24:37.732740 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.732723 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 11:24:37.733143 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.733126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:24:37.740533 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.740514 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh"] Jul 22 11:24:37.740613 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.740547 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p"] Jul 22 11:24:37.740672 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.740654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.742784 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.742763 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:24:37.742878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.742802 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 11:24:37.742878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.742826 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m8nqc\"" Jul 22 11:24:37.743357 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.743323 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 11:24:37.745026 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.745007 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz"] Jul 22 11:24:37.777054 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29pp\" (UniqueName: \"kubernetes.io/projected/22b92f2e-6199-4aea-85cc-33afb97f8ea0-kube-api-access-h29pp\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.777179 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vfx\" (UniqueName: \"kubernetes.io/projected/0344bcd5-3504-482c-af78-891468e3db88-kube-api-access-26vfx\") pod \"volume-data-source-validator-78cf4c6f6-dkv9p\" (UID: \"0344bcd5-3504-482c-af78-891468e3db88\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" Jul 22 11:24:37.777179 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.777179 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b92f2e-6199-4aea-85cc-33afb97f8ea0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.777327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7825724-e23c-4045-b94b-cf26c1c1f6cb-config\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.777365 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777341 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46l97\" (UniqueName: \"kubernetes.io/projected/c7825724-e23c-4045-b94b-cf26c1c1f6cb-kube-api-access-46l97\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.777398 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmlt\" (UniqueName: \"kubernetes.io/projected/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-kube-api-access-cnmlt\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.777449 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7825724-e23c-4045-b94b-cf26c1c1f6cb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.777498 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b92f2e-6199-4aea-85cc-33afb97f8ea0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.777703 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.777685 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b92f2e-6199-4aea-85cc-33afb97f8ea0-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.779905 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.779885 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22b92f2e-6199-4aea-85cc-33afb97f8ea0-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.787492 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.787472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29pp\" (UniqueName: \"kubernetes.io/projected/22b92f2e-6199-4aea-85cc-33afb97f8ea0-kube-api-access-h29pp\") pod \"kube-storage-version-migrator-operator-7658d9b749-d8z7t\" (UID: \"22b92f2e-6199-4aea-85cc-33afb97f8ea0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.818437 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.818413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:37.878942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.878858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vfx\" (UniqueName: \"kubernetes.io/projected/0344bcd5-3504-482c-af78-891468e3db88-kube-api-access-26vfx\") pod \"volume-data-source-validator-78cf4c6f6-dkv9p\" (UID: \"0344bcd5-3504-482c-af78-891468e3db88\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" Jul 22 11:24:37.878942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.878941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.879123 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.879020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7825724-e23c-4045-b94b-cf26c1c1f6cb-config\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.879179 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.879135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46l97\" (UniqueName: \"kubernetes.io/projected/c7825724-e23c-4045-b94b-cf26c1c1f6cb-kube-api-access-46l97\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.879231 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.879175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmlt\" (UniqueName: \"kubernetes.io/projected/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-kube-api-access-cnmlt\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.879231 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.879203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7825724-e23c-4045-b94b-cf26c1c1f6cb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.879738 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.879713 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7825724-e23c-4045-b94b-cf26c1c1f6cb-config\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.879975 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:37.879934 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:24:37.880172 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:37.880151 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls podName:15a972d8-fd3f-401c-b957-9b0e36d9c4d8 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:38.380127497 +0000 UTC m=+48.541670179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-4gqnz" (UID: "15a972d8-fd3f-401c-b957-9b0e36d9c4d8") : secret "samples-operator-tls" not found Jul 22 11:24:37.882505 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.882481 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7825724-e23c-4045-b94b-cf26c1c1f6cb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.889291 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.889268 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmlt\" (UniqueName: \"kubernetes.io/projected/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-kube-api-access-cnmlt\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:37.890572 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.890527 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vfx\" (UniqueName: \"kubernetes.io/projected/0344bcd5-3504-482c-af78-891468e3db88-kube-api-access-26vfx\") pod \"volume-data-source-validator-78cf4c6f6-dkv9p\" (UID: \"0344bcd5-3504-482c-af78-891468e3db88\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" Jul 22 11:24:37.890572 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.890542 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46l97\" (UniqueName: \"kubernetes.io/projected/c7825724-e23c-4045-b94b-cf26c1c1f6cb-kube-api-access-46l97\") pod \"service-ca-operator-5bcbc94c78-b6bkh\" (UID: \"c7825724-e23c-4045-b94b-cf26c1c1f6cb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:37.917327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.917302 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" Jul 22 11:24:37.964552 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:37.964520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gs8kc"] Jul 22 11:24:37.967373 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:37.967349 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40e1c5a_dc8b_4702_a625_6761d7e68330.slice/crio-1454929c75cd3d10c2711b6ce742ac1ccfba37574043d69fcd32901b6ad05669 WatchSource:0}: Error finding container 1454929c75cd3d10c2711b6ce742ac1ccfba37574043d69fcd32901b6ad05669: Status 404 returned error can't find the container with id 1454929c75cd3d10c2711b6ce742ac1ccfba37574043d69fcd32901b6ad05669 Jul 22 11:24:38.027231 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.027195 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" Jul 22 11:24:38.042346 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.042318 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" Jul 22 11:24:38.056069 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.056044 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t"] Jul 22 11:24:38.059413 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:38.059390 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b92f2e_6199_4aea_85cc_33afb97f8ea0.slice/crio-ce51c27d61e89594587746b6993999b95c0d166377d0fa757366d4df0add4957 WatchSource:0}: Error finding container ce51c27d61e89594587746b6993999b95c0d166377d0fa757366d4df0add4957: Status 404 returned error can't find the container with id ce51c27d61e89594587746b6993999b95c0d166377d0fa757366d4df0add4957 Jul 22 11:24:38.175365 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.175333 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh"] Jul 22 11:24:38.177842 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:38.177807 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7825724_e23c_4045_b94b_cf26c1c1f6cb.slice/crio-c45e021f71b4143975e63b4043e3a9c92b98a6179056b5b74dfaf1ce470a462a WatchSource:0}: Error finding container c45e021f71b4143975e63b4043e3a9c92b98a6179056b5b74dfaf1ce470a462a: Status 404 returned error can't find the container with id c45e021f71b4143975e63b4043e3a9c92b98a6179056b5b74dfaf1ce470a462a Jul 22 11:24:38.190688 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.190666 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p"] Jul 22 11:24:38.193611 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:38.193589 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0344bcd5_3504_482c_af78_891468e3db88.slice/crio-7a080adb9ea4c70cacf16c04ac0a7610501e479acf769852cdcdb8e324d3444d WatchSource:0}: Error finding container 7a080adb9ea4c70cacf16c04ac0a7610501e479acf769852cdcdb8e324d3444d: Status 404 returned error can't find the container with id 7a080adb9ea4c70cacf16c04ac0a7610501e479acf769852cdcdb8e324d3444d Jul 22 11:24:38.384444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.384414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:38.384567 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:38.384519 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:24:38.384605 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:38.384577 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls podName:15a972d8-fd3f-401c-b957-9b0e36d9c4d8 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:39.384563786 +0000 UTC m=+49.546106461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-4gqnz" (UID: "15a972d8-fd3f-401c-b957-9b0e36d9c4d8") : secret "samples-operator-tls" not found Jul 22 11:24:38.657101 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.657012 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" event={"ID":"22b92f2e-6199-4aea-85cc-33afb97f8ea0","Type":"ContainerStarted","Data":"ce51c27d61e89594587746b6993999b95c0d166377d0fa757366d4df0add4957"} Jul 22 11:24:38.658388 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.658356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" event={"ID":"c7825724-e23c-4045-b94b-cf26c1c1f6cb","Type":"ContainerStarted","Data":"c45e021f71b4143975e63b4043e3a9c92b98a6179056b5b74dfaf1ce470a462a"} Jul 22 11:24:38.662157 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.662125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" event={"ID":"0344bcd5-3504-482c-af78-891468e3db88","Type":"ContainerStarted","Data":"7a080adb9ea4c70cacf16c04ac0a7610501e479acf769852cdcdb8e324d3444d"} Jul 22 11:24:38.663876 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:38.663852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" event={"ID":"c40e1c5a-dc8b-4702-a625-6761d7e68330","Type":"ContainerStarted","Data":"1454929c75cd3d10c2711b6ce742ac1ccfba37574043d69fcd32901b6ad05669"} Jul 22 11:24:39.394942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:39.394783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:39.395134 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:39.394979 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:24:39.395134 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:39.395057 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls podName:15a972d8-fd3f-401c-b957-9b0e36d9c4d8 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:41.395036132 +0000 UTC m=+51.556578827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-4gqnz" (UID: "15a972d8-fd3f-401c-b957-9b0e36d9c4d8") : secret "samples-operator-tls" not found Jul 22 11:24:40.605512 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:40.605476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:40.605919 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:40.605586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:40.605919 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:40.605650 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:40.605919 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:40.605717 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:24:40.605919 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:40.605721 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:48.605704821 +0000 UTC m=+58.767247494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:40.605919 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:40.605823 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:48.60581039 +0000 UTC m=+58.767353066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : configmap "kube-rbac-proxy" not found Jul 22 11:24:41.411588 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:41.411550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:41.411746 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:41.411670 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:24:41.411746 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:41.411722 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls podName:15a972d8-fd3f-401c-b957-9b0e36d9c4d8 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:45.411708563 +0000 UTC m=+55.573251237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-4gqnz" (UID: "15a972d8-fd3f-401c-b957-9b0e36d9c4d8") : secret "samples-operator-tls" not found Jul 22 11:24:42.118212 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.118177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:42.118570 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:42.118340 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:24:42.118570 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:42.118402 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls podName:0150094a-8a0e-4152-a85f-481cdede5afa nodeName:}" failed. No retries permitted until 2026-07-22 11:24:58.118388197 +0000 UTC m=+68.279930875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls") pod "dns-default-psq2l" (UID: "0150094a-8a0e-4152-a85f-481cdede5afa") : secret "dns-default-metrics-tls" not found Jul 22 11:24:42.677742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.677314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" event={"ID":"c40e1c5a-dc8b-4702-a625-6761d7e68330","Type":"ContainerStarted","Data":"fe0c63b9876d793a88802efc501fda6f97cc0b82d6ed7bd481b6c6dfcf9ca5ac"} Jul 22 11:24:42.677742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.677666 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:42.678815 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.678774 2569 patch_prober.go:28] interesting pod/console-operator-565b7b569d-gs8kc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body= Jul 22 11:24:42.678917 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.678829 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podUID="c40e1c5a-dc8b-4702-a625-6761d7e68330" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused" Jul 22 11:24:42.679527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.679491 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" event={"ID":"c7825724-e23c-4045-b94b-cf26c1c1f6cb","Type":"ContainerStarted","Data":"0357b8a3ab78d24534b40beebf8d318e2a92bad044e124befeac6cd657059471"} Jul 22 11:24:42.700213 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:42.700162 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podStartSLOduration=1.132319238 podStartE2EDuration="5.70014456s" totalImagesPullingTime="4.567825322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:42.69953314 +0000 UTC m=+52.861075868" watchObservedRunningTime="2026-07-22 11:24:42.70014456 +0000 UTC m=+52.861687237" Jul 22 11:24:43.685347 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.685310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" event={"ID":"22b92f2e-6199-4aea-85cc-33afb97f8ea0","Type":"ContainerStarted","Data":"34a254cf62d9c1c0cd6c93a64f83a42d2280abf5c220c1c48cd377c847b0ac03"} Jul 22 11:24:43.687011 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.686984 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" event={"ID":"0344bcd5-3504-482c-af78-891468e3db88","Type":"ContainerStarted","Data":"f776cf1c872b4f3e3c6223ee2811c56fb96e4f5d5caaa42d19744e1dab106f87"} Jul 22 11:24:43.688388 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.688369 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/0.log" Jul 22 11:24:43.688513 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.688402 2569 generic.go:356] "Generic (PLEG): container finished" podID="c40e1c5a-dc8b-4702-a625-6761d7e68330" containerID="fe0c63b9876d793a88802efc501fda6f97cc0b82d6ed7bd481b6c6dfcf9ca5ac" exitCode=255 Jul 22 11:24:43.688513 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.688431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" event={"ID":"c40e1c5a-dc8b-4702-a625-6761d7e68330","Type":"ContainerDied","Data":"fe0c63b9876d793a88802efc501fda6f97cc0b82d6ed7bd481b6c6dfcf9ca5ac"} Jul 22 11:24:43.688743 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.688665 2569 scope.go:117] "RemoveContainer" containerID="fe0c63b9876d793a88802efc501fda6f97cc0b82d6ed7bd481b6c6dfcf9ca5ac" Jul 22 11:24:43.702755 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.702711 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" podStartSLOduration=2.21772274 podStartE2EDuration="6.702692794s" totalImagesPullingTime="4.484970054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:43.701172489 +0000 UTC m=+53.862715199" watchObservedRunningTime="2026-07-22 11:24:43.702692794 +0000 UTC m=+53.864235492" Jul 22 11:24:43.717760 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.717688 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" podStartSLOduration=2.356957499 podStartE2EDuration="6.717674263s" totalImagesPullingTime="4.360716764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:43.716996474 +0000 UTC m=+53.878539195" watchObservedRunningTime="2026-07-22 11:24:43.717674263 +0000 UTC m=+53.879216960" Jul 22 11:24:43.732635 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.732599 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dkv9p" podStartSLOduration=2.387342061 podStartE2EDuration="6.732586553s" totalImagesPullingTime="4.345244492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:43.732074497 +0000 UTC m=+53.893617197" watchObservedRunningTime="2026-07-22 11:24:43.732586553 +0000 UTC m=+53.894129249" Jul 22 11:24:43.827834 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.827805 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:43.934263 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:43.934238 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:44.008111 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.008035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:44.090192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.090147 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:44.378527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.378488 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:44.660389 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.660291 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:44.692354 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.692330 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:24:44.692779 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.692760 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/0.log" Jul 22 11:24:44.692856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.692804 2569 generic.go:356] "Generic (PLEG): container finished" podID="c40e1c5a-dc8b-4702-a625-6761d7e68330" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" exitCode=255 Jul 22 11:24:44.692921 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.692905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" event={"ID":"c40e1c5a-dc8b-4702-a625-6761d7e68330","Type":"ContainerDied","Data":"d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163"} Jul 22 11:24:44.692993 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.692977 2569 scope.go:117] "RemoveContainer" containerID="fe0c63b9876d793a88802efc501fda6f97cc0b82d6ed7bd481b6c6dfcf9ca5ac" Jul 22 11:24:44.693166 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.693139 2569 scope.go:117] "RemoveContainer" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" Jul 22 11:24:44.693346 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:44.693320 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gs8kc_openshift-console-operator(c40e1c5a-dc8b-4702-a625-6761d7e68330)\"" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podUID="c40e1c5a-dc8b-4702-a625-6761d7e68330" Jul 22 11:24:44.752183 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:44.752161 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:45.446211 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:45.446180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:45.446389 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:45.446345 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:24:45.446452 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:45.446421 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls podName:15a972d8-fd3f-401c-b957-9b0e36d9c4d8 nodeName:}" failed. No retries permitted until 2026-07-22 11:24:53.446399795 +0000 UTC m=+63.607942469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-4gqnz" (UID: "15a972d8-fd3f-401c-b957-9b0e36d9c4d8") : secret "samples-operator-tls" not found Jul 22 11:24:45.700014 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:45.699932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:24:45.700454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:45.700350 2569 scope.go:117] "RemoveContainer" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" Jul 22 11:24:45.700547 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:45.700525 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gs8kc_openshift-console-operator(c40e1c5a-dc8b-4702-a625-6761d7e68330)\"" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podUID="c40e1c5a-dc8b-4702-a625-6761d7e68330" Jul 22 11:24:46.063218 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:46.063187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:46.131903 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:46.131871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:46.308534 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:46.308504 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:47.616130 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:47.616092 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:47.758266 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:47.758230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:47.819144 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:47.819116 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:47.819514 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:47.819499 2569 scope.go:117] "RemoveContainer" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" Jul 22 11:24:47.819653 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:47.819635 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gs8kc_openshift-console-operator(c40e1c5a-dc8b-4702-a625-6761d7e68330)\"" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podUID="c40e1c5a-dc8b-4702-a625-6761d7e68330" Jul 22 11:24:48.670978 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:48.670922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:48.671426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:48.671032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:48.671426 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:48.671181 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:24:48.671426 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:48.671266 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls podName:5455f9ae-4a87-468e-8025-adc4622ed566 nodeName:}" failed. No retries permitted until 2026-07-22 11:25:04.671245575 +0000 UTC m=+74.832788270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5bw4" (UID: "5455f9ae-4a87-468e-8025-adc4622ed566") : secret "insights-runtime-extractor-tls" not found Jul 22 11:24:48.671426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:48.671388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5455f9ae-4a87-468e-8025-adc4622ed566-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:24:49.619095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:49.619058 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:49.751555 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:49.751516 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:49.845548 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:49.845511 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:49.949863 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:49.949782 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:50.603489 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:50.603462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5smx8" Jul 22 11:24:50.920649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:50.920558 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:51.277596 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:51.277494 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:51.370307 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:51.370280 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:51.526674 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:51.526644 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:51.723268 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:51.723246 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:52.114361 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:52.114330 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:52.677714 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:52.677688 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:24:52.678175 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:52.678156 2569 scope.go:117] "RemoveContainer" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" Jul 22 11:24:52.678366 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:52.678342 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gs8kc_openshift-console-operator(c40e1c5a-dc8b-4702-a625-6761d7e68330)\"" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" podUID="c40e1c5a-dc8b-4702-a625-6761d7e68330" Jul 22 11:24:53.194237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.194199 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:53.298854 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.298825 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:53.392846 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.392823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:53.509608 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.509545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:53.512254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.512231 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/15a972d8-fd3f-401c-b957-9b0e36d9c4d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-4gqnz\" (UID: \"15a972d8-fd3f-401c-b957-9b0e36d9c4d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:53.524517 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.524495 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:53.652590 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.652566 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m8nqc\"" Jul 22 11:24:53.660628 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.660609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" Jul 22 11:24:53.714679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.714658 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:53.782714 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:53.782654 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz"] Jul 22 11:24:54.110266 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:54.110230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:54.178973 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:54.178923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:54.734290 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:54.734254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" event={"ID":"15a972d8-fd3f-401c-b957-9b0e36d9c4d8","Type":"ContainerStarted","Data":"f17ed2e6ef6f6e8a731a7e87f4abf0488a8331d500d27ccb07d33b4e0c46ee58"} Jul 22 11:24:54.911735 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:54.911683 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:55.114891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:55.114857 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:55.525984 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:55.525863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:55.745315 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:55.745270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:55.884090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:55.884061 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:56.134335 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.134288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:24:56.134335 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.134338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:56.134544 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.134382 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:56.139732 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.139704 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:24:56.139732 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.139725 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:24:56.139852 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.139729 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:24:56.144821 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:56.144792 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 11:24:56.144893 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:24:56.144879 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs podName:d89a345f-130b-45e0-b661-768c66f0fbcb nodeName:}" failed. No retries permitted until 2026-07-22 11:26:00.144854874 +0000 UTC m=+130.306397549 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs") pod "network-metrics-daemon-7qbhz" (UID: "d89a345f-130b-45e0-b661-768c66f0fbcb") : secret "metrics-daemon-secret" not found Jul 22 11:24:56.147346 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.147309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9c9f01e-fd6b-438e-a92c-11fddaaae685-original-pull-secret\") pod \"global-pull-secret-syncer-jwjv7\" (UID: \"d9c9f01e-fd6b-438e-a92c-11fddaaae685\") " pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:56.149883 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.149865 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:24:56.158131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.158102 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qn26\" (UniqueName: \"kubernetes.io/projected/d1d0d74c-4ddf-42e8-84df-82156efc1f2e-kube-api-access-4qn26\") pod \"network-check-target-5wn2g\" (UID: \"d1d0d74c-4ddf-42e8-84df-82156efc1f2e\") " pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:56.162635 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.162594 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:56.358617 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.358588 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7fc2b\"" Jul 22 11:24:56.366694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.366656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:24:56.369720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.369698 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwjv7" Jul 22 11:24:56.514881 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.514852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5wn2g"] Jul 22 11:24:56.517768 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:56.517742 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d0d74c_4ddf_42e8_84df_82156efc1f2e.slice/crio-ab6927b8f3d4f2f2a71439846e39eff0b6d73620485ff2de9411aa1ed1d6ccee WatchSource:0}: Error finding container ab6927b8f3d4f2f2a71439846e39eff0b6d73620485ff2de9411aa1ed1d6ccee: Status 404 returned error can't find the container with id ab6927b8f3d4f2f2a71439846e39eff0b6d73620485ff2de9411aa1ed1d6ccee Jul 22 11:24:56.528469 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.528446 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwjv7"] Jul 22 11:24:56.532413 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:56.532385 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c9f01e_fd6b_438e_a92c_11fddaaae685.slice/crio-b59f4012a9f61aa7d080ab936a69f628d415e82820b07dfa61b7fb01f53bc5ca WatchSource:0}: Error finding container b59f4012a9f61aa7d080ab936a69f628d415e82820b07dfa61b7fb01f53bc5ca: Status 404 returned error can't find the container with id b59f4012a9f61aa7d080ab936a69f628d415e82820b07dfa61b7fb01f53bc5ca Jul 22 11:24:56.742245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.742145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" event={"ID":"15a972d8-fd3f-401c-b957-9b0e36d9c4d8","Type":"ContainerStarted","Data":"777246cb62cfdae06eb83aca99dbb5b1e90580070cd5d2ba93501b7c42f64366"} Jul 22 11:24:56.742245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.742199 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" event={"ID":"15a972d8-fd3f-401c-b957-9b0e36d9c4d8","Type":"ContainerStarted","Data":"fadead99799e5affc31f51933204e95d845552aa35df8dae9102990fb7f8cce1"} Jul 22 11:24:56.743419 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.743391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5wn2g" event={"ID":"d1d0d74c-4ddf-42e8-84df-82156efc1f2e","Type":"ContainerStarted","Data":"ab6927b8f3d4f2f2a71439846e39eff0b6d73620485ff2de9411aa1ed1d6ccee"} Jul 22 11:24:56.744410 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.744389 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwjv7" event={"ID":"d9c9f01e-fd6b-438e-a92c-11fddaaae685","Type":"ContainerStarted","Data":"b59f4012a9f61aa7d080ab936a69f628d415e82820b07dfa61b7fb01f53bc5ca"} Jul 22 11:24:56.760809 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.760754 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-4gqnz" podStartSLOduration=17.812704747 podStartE2EDuration="19.760737186s" totalImagesPullingTime="1.948032439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:24:56.75967429 +0000 UTC m=+66.921216994" watchObservedRunningTime="2026-07-22 11:24:56.760737186 +0000 UTC m=+66.922279892" Jul 22 11:24:56.955509 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:56.955464 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:57.326576 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:57.326535 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:57.570628 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:57.570561 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:57.743682 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:57.743545 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:57.965937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:57.965853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:58.151160 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.150651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:58.153795 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.153733 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0150094a-8a0e-4152-a85f-481cdede5afa-metrics-tls\") pod \"dns-default-psq2l\" (UID: \"0150094a-8a0e-4152-a85f-481cdede5afa\") " pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:58.335289 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.335256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqh28\"" Jul 22 11:24:58.343310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.343284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-psq2l" Jul 22 11:24:58.497939 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.497902 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-psq2l"] Jul 22 11:24:58.501695 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:24:58.501661 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0150094a_8a0e_4152_a85f_481cdede5afa.slice/crio-efa69f4c66206b31074a7a9b749d3620775ad43413cd57c823fcd102b1bf4250 WatchSource:0}: Error finding container efa69f4c66206b31074a7a9b749d3620775ad43413cd57c823fcd102b1bf4250: Status 404 returned error can't find the container with id efa69f4c66206b31074a7a9b749d3620775ad43413cd57c823fcd102b1bf4250 Jul 22 11:24:58.754134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.754049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-psq2l" event={"ID":"0150094a-8a0e-4152-a85f-481cdede5afa","Type":"ContainerStarted","Data":"efa69f4c66206b31074a7a9b749d3620775ad43413cd57c823fcd102b1bf4250"} Jul 22 11:24:58.832264 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.832163 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:58.934348 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:58.934301 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:24:59.350271 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:24:59.350230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:00.479687 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:00.479631 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:00.586071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:00.586035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:02.770077 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:02.770052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwjv7" event={"ID":"d9c9f01e-fd6b-438e-a92c-11fddaaae685","Type":"ContainerStarted","Data":"36c62eca99d7d008672fa3af592f5a2911e7cb277c7a2ccf40640ee5c78e6b05"} Jul 22 11:25:02.771440 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:02.771413 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5wn2g" event={"ID":"d1d0d74c-4ddf-42e8-84df-82156efc1f2e","Type":"ContainerStarted","Data":"ae14e4523cc19c6d41902f968e267187d83aeb3f3a12f6a891f3bfaece75510e"} Jul 22 11:25:02.776557 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:02.776539 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:25:02.793512 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:02.793476 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwjv7" podStartSLOduration=65.93401492 podStartE2EDuration="1m11.793465863s" totalImagesPullingTime="5.859450943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:02.793202376 +0000 UTC m=+72.954745092" watchObservedRunningTime="2026-07-22 11:25:02.793465863 +0000 UTC m=+72.955008558" Jul 22 11:25:02.810502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:02.810461 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5wn2g" podStartSLOduration=65.936344646 podStartE2EDuration="1m11.810449403s" totalImagesPullingTime="5.874104757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:02.809450605 +0000 UTC m=+72.970993321" watchObservedRunningTime="2026-07-22 11:25:02.810449403 +0000 UTC m=+72.971992100" Jul 22 11:25:03.776526 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:03.776486 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-psq2l" event={"ID":"0150094a-8a0e-4152-a85f-481cdede5afa","Type":"ContainerStarted","Data":"36e331ecd2e19ff4bc59d7602b52648172eb1fc79c46d187cb3ec57d2aa8f32e"} Jul 22 11:25:03.777106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:03.776551 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-psq2l" event={"ID":"0150094a-8a0e-4152-a85f-481cdede5afa","Type":"ContainerStarted","Data":"f6ff146a8d8020daab490848ac4871a159ef74d8779e6a3e767a0201cb489208"} Jul 22 11:25:03.797896 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:03.797845 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-psq2l" podStartSLOduration=33.548799088 podStartE2EDuration="37.797828358s" totalImagesPullingTime="4.24902927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:03.796861034 +0000 UTC m=+73.958403730" watchObservedRunningTime="2026-07-22 11:25:03.797828358 +0000 UTC m=+73.959371104" Jul 22 11:25:04.702404 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.702368 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:25:04.703339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.703308 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:04.705108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.705085 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5455f9ae-4a87-468e-8025-adc4622ed566-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5bw4\" (UID: \"5455f9ae-4a87-468e-8025-adc4622ed566\") " pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:25:04.752963 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.752922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pxx7\"" Jul 22 11:25:04.760095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.760077 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5bw4" Jul 22 11:25:04.780112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.780091 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-psq2l" Jul 22 11:25:04.808236 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.807920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:04.893654 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.893630 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5bw4"] Jul 22 11:25:04.896327 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:25:04.896301 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5455f9ae_4a87_468e_8025_adc4622ed566.slice/crio-686b7577f5c2f810aaf04bf95f74f93123edf290ed002697c8388493db9f61f2 WatchSource:0}: Error finding container 686b7577f5c2f810aaf04bf95f74f93123edf290ed002697c8388493db9f61f2: Status 404 returned error can't find the container with id 686b7577f5c2f810aaf04bf95f74f93123edf290ed002697c8388493db9f61f2 Jul 22 11:25:04.900673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:04.900628 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:05.087303 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:05.087275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:05.786019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:05.785973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5bw4" event={"ID":"5455f9ae-4a87-468e-8025-adc4622ed566","Type":"ContainerStarted","Data":"686b7577f5c2f810aaf04bf95f74f93123edf290ed002697c8388493db9f61f2"} Jul 22 11:25:06.349061 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.349028 2569 scope.go:117] "RemoveContainer" containerID="d508978aa2b6ad2dc1f8a87ac967921f0bb20ab082ba6df4140b7422bf2df163" Jul 22 11:25:06.610659 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.610562 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8"] Jul 22 11:25:06.613375 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.613349 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.633976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.629144 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 11:25:06.633976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.629392 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 11:25:06.633976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.629565 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 11:25:06.633976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.629842 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 11:25:06.634310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.634293 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 11:25:06.638682 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.638649 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 11:25:06.638831 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.638811 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 11:25:06.642088 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.642064 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8"] Jul 22 11:25:06.729498 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.729498 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcc5\" (UniqueName: \"kubernetes.io/projected/e1adf6a8-6e1e-48db-b98d-fedd510efc78-kube-api-access-2gcc5\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.729723 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1adf6a8-6e1e-48db-b98d-fedd510efc78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.729723 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.729723 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729603 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.729723 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.729637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.794130 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.794095 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:25:06.794574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.794243 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" event={"ID":"c40e1c5a-dc8b-4702-a625-6761d7e68330","Type":"ContainerStarted","Data":"ddbbfe02a0b4450674b743db2d666ed1cd1c46222d26908301036b8400525bb2"} Jul 22 11:25:06.794804 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.794780 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:25:06.830052 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.830052 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcc5\" (UniqueName: \"kubernetes.io/projected/e1adf6a8-6e1e-48db-b98d-fedd510efc78-kube-api-access-2gcc5\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.830294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1adf6a8-6e1e-48db-b98d-fedd510efc78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.830294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.830294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.830294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.830213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.831513 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.831453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1adf6a8-6e1e-48db-b98d-fedd510efc78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.833986 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.833802 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-ca\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.834141 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.834115 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.834255 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.834161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.834945 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.834913 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1adf6a8-6e1e-48db-b98d-fedd510efc78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.850466 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.850435 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcc5\" (UniqueName: \"kubernetes.io/projected/e1adf6a8-6e1e-48db-b98d-fedd510efc78-kube-api-access-2gcc5\") pod \"cluster-proxy-proxy-agent-68d7bfc777-vgcz8\" (UID: \"e1adf6a8-6e1e-48db-b98d-fedd510efc78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.853089 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.853054 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:06.955695 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.955608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" Jul 22 11:25:06.983870 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:06.983833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:07.080758 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.080726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:07.122588 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.122552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8"] Jul 22 11:25:07.126546 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:25:07.126518 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1adf6a8_6e1e_48db_b98d_fedd510efc78.slice/crio-1329ce41743bc8c1cbcbcccc4cf3cc3b9e5fc898d40fb0b3594b0afd535fdb19 WatchSource:0}: Error finding container 1329ce41743bc8c1cbcbcccc4cf3cc3b9e5fc898d40fb0b3594b0afd535fdb19: Status 404 returned error can't find the container with id 1329ce41743bc8c1cbcbcccc4cf3cc3b9e5fc898d40fb0b3594b0afd535fdb19 Jul 22 11:25:07.578425 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.578393 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-gs8kc" Jul 22 11:25:07.793815 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.793780 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:07.799358 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.799328 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" event={"ID":"e1adf6a8-6e1e-48db-b98d-fedd510efc78","Type":"ContainerStarted","Data":"1329ce41743bc8c1cbcbcccc4cf3cc3b9e5fc898d40fb0b3594b0afd535fdb19"} Jul 22 11:25:07.903465 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.903389 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:07.992640 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:07.992586 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:09.323264 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:09.323231 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:09.400513 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:09.400455 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:09.483762 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:09.483724 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:09.684418 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:09.684328 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:10.817834 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:10.817706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5bw4" event={"ID":"5455f9ae-4a87-468e-8025-adc4622ed566","Type":"ContainerStarted","Data":"2b902b4a1df8e884d28ebbc7daefb892390ad96746088b82585ef52b19e8cdc1"} Jul 22 11:25:11.086478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:11.086390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:11.160875 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:11.160835 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:14.788799 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:14.788771 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-psq2l" Jul 22 11:25:15.366151 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.366118 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:15.498049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.498006 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:15.652910 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.652884 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:15.838141 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.838106 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5bw4" event={"ID":"5455f9ae-4a87-468e-8025-adc4622ed566","Type":"ContainerStarted","Data":"067837fa0fe7066412fd06dd3ffdccf9e5d06c61f82c897753c1520e85cc5447"} Jul 22 11:25:15.840237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.840216 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" event={"ID":"e1adf6a8-6e1e-48db-b98d-fedd510efc78","Type":"ContainerStarted","Data":"3ecbd12b27bf7ebc25421d2cf07318ebad731b69b0a72a6c00f6f526e2a24c1c"} Jul 22 11:25:15.840320 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.840254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" event={"ID":"e1adf6a8-6e1e-48db-b98d-fedd510efc78","Type":"ContainerStarted","Data":"c2cd95c6e6d2a3870a4476515b6f83c6d1fd702bcbfebe6d8250fe996f52e630"} Jul 22 11:25:15.840320 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.840277 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" event={"ID":"e1adf6a8-6e1e-48db-b98d-fedd510efc78","Type":"ContainerStarted","Data":"92662fabccaf9366a248ac35f9931fdaa75eb018f81b372edc5ee14e934f871d"} Jul 22 11:25:15.860647 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.860582 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d7bfc777-vgcz8" podStartSLOduration=1.46391267 podStartE2EDuration="9.860567717s" totalImagesPullingTime="8.396655047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:15.860020978 +0000 UTC m=+86.021563674" watchObservedRunningTime="2026-07-22 11:25:15.860567717 +0000 UTC m=+86.022110465" Jul 22 11:25:15.996845 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:15.996781 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:16.282160 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:16.282088 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:16.413828 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:16.413795 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:16.899597 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:16.899556 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:17.050810 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:17.050768 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:18.332884 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:18.332846 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:19.298969 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:19.298912 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:19.406793 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:19.406759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:19.519300 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:19.519263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:19.856197 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:19.856161 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5bw4" event={"ID":"5455f9ae-4a87-468e-8025-adc4622ed566","Type":"ContainerStarted","Data":"fe661cfefa8299e4fe0b88c3edb95dcc960a648a2310e9bdcc300d4bffafa053"} Jul 22 11:25:19.874991 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:19.874902 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m5bw4" podStartSLOduration=33.562184732 podStartE2EDuration="47.874885779s" totalImagesPullingTime="14.312701047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:24:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:19.874667484 +0000 UTC m=+90.036210179" watchObservedRunningTime="2026-07-22 11:25:19.874885779 +0000 UTC m=+90.036428469" Jul 22 11:25:20.347606 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.347566 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.472771 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.472726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.620487 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.620403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.674232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.674199 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g8bq6"] Jul 22 11:25:20.676499 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.676471 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.679873 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.679846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 11:25:20.680020 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.679897 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 11:25:20.680449 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.680426 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 11:25:20.680449 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.680441 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 11:25:20.680568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.680465 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 11:25:20.680568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.680506 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nfv49\"" Jul 22 11:25:20.680673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.680565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 11:25:20.735580 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.735541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.745301 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745264 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745301 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-tls\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-root\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-sys\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp4r\" (UniqueName: \"kubernetes.io/projected/8284171d-cec2-4e37-af8c-3edd06ebbfb7-kube-api-access-vpp4r\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-textfile\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745671 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-wtmp\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.745671 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.745540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-metrics-client-ca\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.827399 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.827367 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.846009 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.845975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-root\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846017 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-sys\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-root\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846038 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp4r\" (UniqueName: \"kubernetes.io/projected/8284171d-cec2-4e37-af8c-3edd06ebbfb7-kube-api-access-vpp4r\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-textfile\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846108 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-sys\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-wtmp\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-metrics-client-ca\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-tls\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846282 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-wtmp\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846515 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846515 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846389 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-textfile\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846746 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846720 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.846848 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.846815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8284171d-cec2-4e37-af8c-3edd06ebbfb7-metrics-client-ca\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.848780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.848753 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-tls\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.848880 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.848784 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8284171d-cec2-4e37-af8c-3edd06ebbfb7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.855582 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.855558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp4r\" (UniqueName: \"kubernetes.io/projected/8284171d-cec2-4e37-af8c-3edd06ebbfb7-kube-api-access-vpp4r\") pod \"node-exporter-g8bq6\" (UID: \"8284171d-cec2-4e37-af8c-3edd06ebbfb7\") " pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.926015 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.925905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:20.986175 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:20.986139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8bq6" Jul 22 11:25:20.997108 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:25:20.997077 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8284171d_cec2_4e37_af8c_3edd06ebbfb7.slice/crio-f42d35f351804bccb84b50f09786007a859feacb956ea9ae301b7fcb85fe2b02 WatchSource:0}: Error finding container f42d35f351804bccb84b50f09786007a859feacb956ea9ae301b7fcb85fe2b02: Status 404 returned error can't find the container with id f42d35f351804bccb84b50f09786007a859feacb956ea9ae301b7fcb85fe2b02 Jul 22 11:25:21.116242 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.116200 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:21.220500 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.220401 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:21.448805 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.448768 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:21.590443 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.590399 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:21.691931 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.691880 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:21.864012 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.863919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8bq6" event={"ID":"8284171d-cec2-4e37-af8c-3edd06ebbfb7","Type":"ContainerStarted","Data":"9928cfeef089cd85e8f9e4320f88b2108713d8ecf2ade03153f7d622158fb67c"} Jul 22 11:25:21.864012 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:21.864000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8bq6" event={"ID":"8284171d-cec2-4e37-af8c-3edd06ebbfb7","Type":"ContainerStarted","Data":"f42d35f351804bccb84b50f09786007a859feacb956ea9ae301b7fcb85fe2b02"} Jul 22 11:25:22.869739 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:22.869706 2569 generic.go:356] "Generic (PLEG): container finished" podID="8284171d-cec2-4e37-af8c-3edd06ebbfb7" containerID="9928cfeef089cd85e8f9e4320f88b2108713d8ecf2ade03153f7d622158fb67c" exitCode=0 Jul 22 11:25:22.870161 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:22.869749 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8bq6" event={"ID":"8284171d-cec2-4e37-af8c-3edd06ebbfb7","Type":"ContainerDied","Data":"9928cfeef089cd85e8f9e4320f88b2108713d8ecf2ade03153f7d622158fb67c"} Jul 22 11:25:23.562975 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:23.562927 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:23.665158 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:23.665111 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:23.807338 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:23.807296 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:23.875214 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:23.875131 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8bq6" event={"ID":"8284171d-cec2-4e37-af8c-3edd06ebbfb7","Type":"ContainerStarted","Data":"cb361dfb12a24117f8cd2c53bd801803064d4101e6776451ef7bd1e9170ee95e"} Jul 22 11:25:23.875214 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:23.875193 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8bq6" event={"ID":"8284171d-cec2-4e37-af8c-3edd06ebbfb7","Type":"ContainerStarted","Data":"4246c5ac714dd08f80afe62b691e9aca64ecf5e417ede56950a737328cdabf60"} Jul 22 11:25:24.003906 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:24.003869 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:24.088110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:24.088082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:24.187571 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:24.187502 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:25.241450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:25.241412 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:25.429773 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:25.429742 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:25.530204 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:25.530131 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:26.301260 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:26.301228 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:26.900389 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:26.900356 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:27.478323 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:27.478295 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:27.578543 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:27.578515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:28.351049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.351022 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:28.434337 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.434300 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:28.691745 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.691659 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g8bq6" podStartSLOduration=7.830308385 podStartE2EDuration="8.691643631s" totalImagesPullingTime="785.275234ms" totalInitContainerRuntime="76.060012ms" isStatefulPod=true podCreationTimestamp="2026-07-22 11:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:23.8984874 +0000 UTC m=+94.060030117" watchObservedRunningTime="2026-07-22 11:25:28.691643631 +0000 UTC m=+98.853186327" Jul 22 11:25:28.692214 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.692198 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:25:28.695406 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.695391 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.697734 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.697700 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 11:25:28.697854 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.697765 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 11:25:28.698242 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.698219 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 11:25:28.698301 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.698219 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 11:25:28.698361 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.698323 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 11:25:28.698573 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.698559 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 11:25:28.699199 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.699184 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 11:25:28.699323 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.699217 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sqhjl\"" Jul 22 11:25:28.702889 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.702703 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 11:25:28.706548 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.706527 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:25:28.799577 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.796331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.799577 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.796463 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897178 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9f59\" (UniqueName: \"kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897332 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897332 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897332 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897253 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.897481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.897454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.899726 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.899703 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.899910 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.899890 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.998666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.998614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9f59\" (UniqueName: \"kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.998666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.998648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.998820 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.998674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.998820 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.998742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.998820 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.998770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.999406 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.999384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.999453 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.999437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.999554 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.999530 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:28.999647 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:28.999631 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:29.008942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.008923 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9f59\" (UniqueName: \"kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59\") pod \"console-f89ddb88f-brcl6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:29.305662 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.305640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:29.446790 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.446762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:25:29.449875 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:25:29.449834 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212fa5d_76b1_460c_9b71_1b1604f681e6.slice/crio-adce77fd056d7aee9fa463f952817b20f340f482e9e3c3d43c95b8fb11ffea89 WatchSource:0}: Error finding container adce77fd056d7aee9fa463f952817b20f340f482e9e3c3d43c95b8fb11ffea89: Status 404 returned error can't find the container with id adce77fd056d7aee9fa463f952817b20f340f482e9e3c3d43c95b8fb11ffea89 Jul 22 11:25:29.467260 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.467237 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:29.550657 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.550630 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:29.896237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:29.896204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f89ddb88f-brcl6" event={"ID":"4212fa5d-76b1-460c-9b71-1b1604f681e6","Type":"ContainerStarted","Data":"adce77fd056d7aee9fa463f952817b20f340f482e9e3c3d43c95b8fb11ffea89"} Jul 22 11:25:30.675673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:30.675637 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:30.777241 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:30.777202 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:31.155894 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:31.155852 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:31.227940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:31.227910 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:32.909085 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:32.909043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f89ddb88f-brcl6" event={"ID":"4212fa5d-76b1-460c-9b71-1b1604f681e6","Type":"ContainerStarted","Data":"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d"} Jul 22 11:25:32.929825 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:32.929776 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f89ddb88f-brcl6" podStartSLOduration=2.234329929 podStartE2EDuration="4.929761808s" totalImagesPullingTime="2.695431879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:25:32.928353165 +0000 UTC m=+103.089895882" watchObservedRunningTime="2026-07-22 11:25:32.929761808 +0000 UTC m=+103.091304504" Jul 22 11:25:33.779435 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:33.779413 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5wn2g" Jul 22 11:25:38.040147 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:38.040103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:38.264518 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:38.264472 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:39.306163 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.306132 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:39.306542 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.306189 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:39.310780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.310758 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:39.315587 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.315561 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:39.411446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.411423 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:39.938074 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:39.938043 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:25:41.516324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:41.516294 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:41.595252 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:41.595216 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:43.035649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:43.035616 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:43.137781 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:43.137751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:43.889469 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:43.889438 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:45.287134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:45.287100 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:46.235781 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:46.235742 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:46.335732 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:46.335694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:46.410178 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:46.410157 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:48.966309 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:48.966276 2569 generic.go:356] "Generic (PLEG): container finished" podID="22b92f2e-6199-4aea-85cc-33afb97f8ea0" containerID="34a254cf62d9c1c0cd6c93a64f83a42d2280abf5c220c1c48cd377c847b0ac03" exitCode=0 Jul 22 11:25:48.966654 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:48.966349 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" event={"ID":"22b92f2e-6199-4aea-85cc-33afb97f8ea0","Type":"ContainerDied","Data":"34a254cf62d9c1c0cd6c93a64f83a42d2280abf5c220c1c48cd377c847b0ac03"} Jul 22 11:25:48.966695 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:48.966652 2569 scope.go:117] "RemoveContainer" containerID="34a254cf62d9c1c0cd6c93a64f83a42d2280abf5c220c1c48cd377c847b0ac03" Jul 22 11:25:49.971616 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:49.971580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d8z7t" event={"ID":"22b92f2e-6199-4aea-85cc-33afb97f8ea0","Type":"ContainerStarted","Data":"15ee128bf0b7c9e481844d98d59a1eddfeadb20938fc4996b614d7bc0b7b43ea"} Jul 22 11:25:53.003700 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:53.003663 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:53.105383 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:53.105352 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:25:53.987021 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:53.986984 2569 generic.go:356] "Generic (PLEG): container finished" podID="c7825724-e23c-4045-b94b-cf26c1c1f6cb" containerID="0357b8a3ab78d24534b40beebf8d318e2a92bad044e124befeac6cd657059471" exitCode=0 Jul 22 11:25:53.987163 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:53.987052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" event={"ID":"c7825724-e23c-4045-b94b-cf26c1c1f6cb","Type":"ContainerDied","Data":"0357b8a3ab78d24534b40beebf8d318e2a92bad044e124befeac6cd657059471"} Jul 22 11:25:53.987424 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:53.987407 2569 scope.go:117] "RemoveContainer" containerID="0357b8a3ab78d24534b40beebf8d318e2a92bad044e124befeac6cd657059471" Jul 22 11:25:54.991429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:25:54.991399 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b6bkh" event={"ID":"c7825724-e23c-4045-b94b-cf26c1c1f6cb","Type":"ContainerStarted","Data":"aee22a6921790e0e889fb4496d023d19e54d5145e2d95c5e92ebb23137d0a23f"} Jul 22 11:26:00.220917 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:00.220877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:26:00.223579 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:00.223554 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a345f-130b-45e0-b661-768c66f0fbcb-metrics-certs\") pod \"network-metrics-daemon-7qbhz\" (UID: \"d89a345f-130b-45e0-b661-768c66f0fbcb\") " pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:26:00.265653 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:00.265622 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs9j5\"" Jul 22 11:26:00.273433 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:00.273412 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qbhz" Jul 22 11:26:00.408141 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:00.408118 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qbhz"] Jul 22 11:26:00.410524 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:26:00.410494 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a345f_130b_45e0_b661_768c66f0fbcb.slice/crio-bbcf37a13b7571231d88b183fcb430c2ecae9546e2cc6bee16b119387b8b9dab WatchSource:0}: Error finding container bbcf37a13b7571231d88b183fcb430c2ecae9546e2cc6bee16b119387b8b9dab: Status 404 returned error can't find the container with id bbcf37a13b7571231d88b183fcb430c2ecae9546e2cc6bee16b119387b8b9dab Jul 22 11:26:01.011242 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:01.011202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qbhz" event={"ID":"d89a345f-130b-45e0-b661-768c66f0fbcb","Type":"ContainerStarted","Data":"bbcf37a13b7571231d88b183fcb430c2ecae9546e2cc6bee16b119387b8b9dab"} Jul 22 11:26:03.020506 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.020474 2569 generic.go:356] "Generic (PLEG): container finished" podID="96174ad0-30ee-46ab-9e18-8eafb79af050" containerID="1232fcb1973f11b55653211386d2ac4241f6222b92a8eccd0c03d2138913eaa4" exitCode=0 Jul 22 11:26:03.020902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.020550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" event={"ID":"96174ad0-30ee-46ab-9e18-8eafb79af050","Type":"ContainerDied","Data":"1232fcb1973f11b55653211386d2ac4241f6222b92a8eccd0c03d2138913eaa4"} Jul 22 11:26:03.020988 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.020900 2569 scope.go:117] "RemoveContainer" containerID="1232fcb1973f11b55653211386d2ac4241f6222b92a8eccd0c03d2138913eaa4" Jul 22 11:26:03.022671 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.022646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qbhz" event={"ID":"d89a345f-130b-45e0-b661-768c66f0fbcb","Type":"ContainerStarted","Data":"7564aa6501392d9e8c293755a572dab1799cb4d5c3831ee12e66d37b191e359b"} Jul 22 11:26:03.022768 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.022694 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qbhz" event={"ID":"d89a345f-130b-45e0-b661-768c66f0fbcb","Type":"ContainerStarted","Data":"c47f71a453a72a42ab084c273f92e1079c8f22a3ba668548d07a700468d15591"} Jul 22 11:26:03.057105 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.057063 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qbhz" podStartSLOduration=130.063327706 podStartE2EDuration="2m12.057050308s" totalImagesPullingTime="1.993722602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:26:03.055671322 +0000 UTC m=+133.217214017" watchObservedRunningTime="2026-07-22 11:26:03.057050308 +0000 UTC m=+133.218593040" Jul 22 11:26:03.236769 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.236737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:03.333719 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:03.333691 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:04.028381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:04.028343 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-g4rwm" event={"ID":"96174ad0-30ee-46ab-9e18-8eafb79af050","Type":"ContainerStarted","Data":"d8225afd38f295e8e69a6738a6ce166ccffb4308458333551c509a2865da6364"} Jul 22 11:26:08.005908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:08.005869 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:08.092228 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:08.092199 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:39.973710 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:39.973659 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:40.251004 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:40.250898 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:50.612970 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:50.612920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:50.745819 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:50.745792 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:50.812036 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:50.812007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:51.357880 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:51.357846 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:51.428975 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:51.428931 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:26:54.884331 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:26:54.884291 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:01.925807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:01.925772 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:27:02.079156 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:02.079120 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:02.181837 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:02.181755 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:02.282650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:02.282611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:07.440473 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:07.440434 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:26.947457 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:26.947396 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f89ddb88f-brcl6" podUID="4212fa5d-76b1-460c-9b71-1b1604f681e6" containerName="console" containerID="cri-o://820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d" gracePeriod=15 Jul 22 11:27:27.189344 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.189321 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:27:27.198437 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198379 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198437 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198427 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198615 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198444 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9f59\" (UniqueName: \"kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198615 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198528 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198615 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198573 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198772 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198627 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198772 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198668 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config\") pod \"4212fa5d-76b1-460c-9b71-1b1604f681e6\" (UID: \"4212fa5d-76b1-460c-9b71-1b1604f681e6\") " Jul 22 11:27:27.198888 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198810 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:27:27.198937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.198840 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config" (OuterVolumeSpecName: "console-config") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:27:27.199039 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.199012 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:27:27.199103 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.199059 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:27:27.200791 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.200764 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:27:27.200883 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.200848 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:27:27.200923 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.200890 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59" (OuterVolumeSpecName: "kube-api-access-p9f59") pod "4212fa5d-76b1-460c-9b71-1b1604f681e6" (UID: "4212fa5d-76b1-460c-9b71-1b1604f681e6"). InnerVolumeSpecName "kube-api-access-p9f59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:27:27.299322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299282 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-oauth-config\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299312 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-trusted-ca-bundle\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299321 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-serving-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299330 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9f59\" (UniqueName: \"kubernetes.io/projected/4212fa5d-76b1-460c-9b71-1b1604f681e6-kube-api-access-p9f59\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299578 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299342 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-console-config\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299578 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299350 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.299578 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.299357 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4212fa5d-76b1-460c-9b71-1b1604f681e6-oauth-serving-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:27:27.331765 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.331732 2569 generic.go:356] "Generic (PLEG): container finished" podID="4212fa5d-76b1-460c-9b71-1b1604f681e6" containerID="820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d" exitCode=0 Jul 22 11:27:27.331891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.331776 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f89ddb88f-brcl6" Jul 22 11:27:27.331891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.331805 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f89ddb88f-brcl6" event={"ID":"4212fa5d-76b1-460c-9b71-1b1604f681e6","Type":"ContainerDied","Data":"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d"} Jul 22 11:27:27.331891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.331851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f89ddb88f-brcl6" event={"ID":"4212fa5d-76b1-460c-9b71-1b1604f681e6","Type":"ContainerDied","Data":"adce77fd056d7aee9fa463f952817b20f340f482e9e3c3d43c95b8fb11ffea89"} Jul 22 11:27:27.331891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.331870 2569 scope.go:117] "RemoveContainer" containerID="820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d" Jul 22 11:27:27.340834 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.340808 2569 scope.go:117] "RemoveContainer" containerID="820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d" Jul 22 11:27:27.341113 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:27:27.341092 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d\": container with ID starting with 820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d not found: ID does not exist" containerID="820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d" Jul 22 11:27:27.341164 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.341123 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d"} err="failed to get container status \"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d\": rpc error: code = NotFound desc = could not find container \"820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d\": container with ID starting with 820ddd2e775d1c2760fcaf8007307723c2c177329b8c95e41cf18ce553a5389d not found: ID does not exist" Jul 22 11:27:27.372839 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.372810 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:27:27.382340 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:27.382318 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f89ddb88f-brcl6"] Jul 22 11:27:28.348316 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:28.348283 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4212fa5d-76b1-460c-9b71-1b1604f681e6" path="/var/lib/kubelet/pods/4212fa5d-76b1-460c-9b71-1b1604f681e6/volumes" Jul 22 11:27:34.072261 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:34.072214 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:34.213277 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:34.213223 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:55.226457 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:55.226418 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:27:55.297418 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:27:55.297380 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:06.957585 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.957549 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:28:06.958045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.957778 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4212fa5d-76b1-460c-9b71-1b1604f681e6" containerName="console" Jul 22 11:28:06.958045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.957788 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4212fa5d-76b1-460c-9b71-1b1604f681e6" containerName="console" Jul 22 11:28:06.958045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.957835 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4212fa5d-76b1-460c-9b71-1b1604f681e6" containerName="console" Jul 22 11:28:06.961684 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.961659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:06.964228 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.964206 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 11:28:06.964361 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.964231 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 11:28:06.964999 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.964978 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 11:28:06.965095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.965039 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 11:28:06.965157 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.964987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 11:28:06.965270 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.965253 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sqhjl\"" Jul 22 11:28:06.965609 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.965586 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 11:28:06.965727 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.965686 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 11:28:06.970342 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.970327 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 11:28:06.976714 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:06.976694 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:28:07.048541 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.048497 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:07.062458 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062430 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062550 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062550 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062550 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062550 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062677 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.062677 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.062599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfgj\" (UniqueName: \"kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.157455 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.157429 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:07.163787 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.163869 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.163869 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163812 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.163869 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.163869 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.164087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfgj\" (UniqueName: \"kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.164087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.163986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.164596 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.164575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.164728 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.164662 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.164790 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.164726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.165019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.164998 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.166482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.166462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.166563 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.166484 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.174073 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.174051 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfgj\" (UniqueName: \"kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj\") pod \"console-6b5795dcf7-gl2xt\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.271635 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.271584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:07.288419 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.288394 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:07.410307 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.410277 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:28:07.413359 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:28:07.413336 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb9afd_85f3_492c_92c4_34f5ecdc85e4.slice/crio-2963be56b97f5af1a141f69652952b2c2dd84b9fc407a6481ed26852ff5711e2 WatchSource:0}: Error finding container 2963be56b97f5af1a141f69652952b2c2dd84b9fc407a6481ed26852ff5711e2: Status 404 returned error can't find the container with id 2963be56b97f5af1a141f69652952b2c2dd84b9fc407a6481ed26852ff5711e2 Jul 22 11:28:07.418883 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.418858 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:07.464665 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.464641 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5795dcf7-gl2xt" event={"ID":"6abb9afd-85f3-492c-92c4-34f5ecdc85e4","Type":"ContainerStarted","Data":"2963be56b97f5af1a141f69652952b2c2dd84b9fc407a6481ed26852ff5711e2"} Jul 22 11:28:07.762330 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.762278 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:07.851780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:07.851748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:08.469774 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:08.469743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5795dcf7-gl2xt" event={"ID":"6abb9afd-85f3-492c-92c4-34f5ecdc85e4","Type":"ContainerStarted","Data":"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514"} Jul 22 11:28:08.489853 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:08.489811 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b5795dcf7-gl2xt" podStartSLOduration=2.489796443 podStartE2EDuration="2.489796443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:28:08.489517398 +0000 UTC m=+258.651060117" watchObservedRunningTime="2026-07-22 11:28:08.489796443 +0000 UTC m=+258.651339139" Jul 22 11:28:17.271757 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.271668 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:17.271757 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.271750 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:17.276450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.276426 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:17.503085 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.503059 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:28:17.706340 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.706288 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:17.806366 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.806301 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:17.903322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:17.903284 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:18.893949 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:18.893916 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:23.008597 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:23.008556 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:23.108918 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:23.108867 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:23.185646 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:23.185621 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:28:50.191551 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:50.191522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:28:50.197533 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:50.197505 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:28:50.203345 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:28:50.203318 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 11:29:02.272796 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:02.272741 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:03.799403 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:03.799375 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:24.582987 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:24.582935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:24.909751 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:24.909658 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:28.363500 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:28.363461 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:28.430222 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:28.430184 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:29.821541 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:29.821509 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:29.935095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:29.935057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:30.041149 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:30.041047 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:32.189327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:32.189292 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:32.409904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:32.409868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:35.360856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:35.360802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:35.424696 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:35.424655 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:39.442245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:39.442205 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:40.353902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.353826 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jg4mm"] Jul 22 11:29:40.356771 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.356754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.359635 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.359613 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 11:29:40.360309 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.360265 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 11:29:40.360565 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.360365 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rp6px\"" Jul 22 11:29:40.367370 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.367342 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jg4mm"] Jul 22 11:29:40.433674 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.433640 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-bound-sa-token\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.433863 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.433683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29p4\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-kube-api-access-v29p4\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.534368 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.534329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-bound-sa-token\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.534760 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.534376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29p4\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-kube-api-access-v29p4\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.545245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.545214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-bound-sa-token\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.545474 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.545451 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29p4\" (UniqueName: \"kubernetes.io/projected/4fded015-fa80-4bcf-8512-fab00f1f4f97-kube-api-access-v29p4\") pod \"cert-manager-55b4d954d5-jg4mm\" (UID: \"4fded015-fa80-4bcf-8512-fab00f1f4f97\") " pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.671273 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.671173 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jg4mm" Jul 22 11:29:40.804977 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.804938 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jg4mm"] Jul 22 11:29:40.807450 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:29:40.807424 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fded015_fa80_4bcf_8512_fab00f1f4f97.slice/crio-7f71e0bef593f24bf2835dc37193bebd559e8465433b90d3d7bb3aba8d15cbd6 WatchSource:0}: Error finding container 7f71e0bef593f24bf2835dc37193bebd559e8465433b90d3d7bb3aba8d15cbd6: Status 404 returned error can't find the container with id 7f71e0bef593f24bf2835dc37193bebd559e8465433b90d3d7bb3aba8d15cbd6 Jul 22 11:29:40.809387 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:40.809369 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:29:41.788893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:41.788839 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jg4mm" event={"ID":"4fded015-fa80-4bcf-8512-fab00f1f4f97","Type":"ContainerStarted","Data":"7f71e0bef593f24bf2835dc37193bebd559e8465433b90d3d7bb3aba8d15cbd6"} Jul 22 11:29:42.449263 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:42.449169 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:42.548618 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:42.548576 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:43.916643 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.916611 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc"] Jul 22 11:29:43.921274 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.921258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:43.923341 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.923319 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:29:43.923864 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.923849 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 11:29:43.923928 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.923888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4zgz7\"" Jul 22 11:29:43.929201 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.929179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc"] Jul 22 11:29:43.964406 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.964380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:43.964528 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:43.964443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2twx\" (UniqueName: \"kubernetes.io/projected/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-kube-api-access-j2twx\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.065107 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.065079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2twx\" (UniqueName: \"kubernetes.io/projected/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-kube-api-access-j2twx\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.065255 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.065126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.065526 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.065469 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.074985 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.074937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2twx\" (UniqueName: \"kubernetes.io/projected/2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048-kube-api-access-j2twx\") pod \"openshift-lws-operator-fd8ccff4c-n98hc\" (UID: \"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.231016 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.230930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" Jul 22 11:29:44.361249 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.361216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc"] Jul 22 11:29:44.364218 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:29:44.364191 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4949a1_2b9d_4f55_9fd9_bf5ddc4f5048.slice/crio-986a18fa2d9db86ade40c59c935791a9061ec2da0d6b2259f11fafb506e2818d WatchSource:0}: Error finding container 986a18fa2d9db86ade40c59c935791a9061ec2da0d6b2259f11fafb506e2818d: Status 404 returned error can't find the container with id 986a18fa2d9db86ade40c59c935791a9061ec2da0d6b2259f11fafb506e2818d Jul 22 11:29:44.801479 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.801446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jg4mm" event={"ID":"4fded015-fa80-4bcf-8512-fab00f1f4f97","Type":"ContainerStarted","Data":"03685877dc4186c53fc418f63212ac7701b27654697f2681036d0bfbe0073d72"} Jul 22 11:29:44.802744 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:44.802716 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" event={"ID":"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048","Type":"ContainerStarted","Data":"986a18fa2d9db86ade40c59c935791a9061ec2da0d6b2259f11fafb506e2818d"} Jul 22 11:29:47.826584 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:47.826547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" event={"ID":"2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048","Type":"ContainerStarted","Data":"27fff1221b689399cae1697f117535ff832ddb386c06dafa3a06c97d8ca8dfe4"} Jul 22 11:29:47.845259 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:47.845199 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jg4mm" podStartSLOduration=4.665664716 podStartE2EDuration="7.845179619s" totalImagesPullingTime="3.179514903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:29:44.827138233 +0000 UTC m=+354.988680928" watchObservedRunningTime="2026-07-22 11:29:47.845179619 +0000 UTC m=+358.006722319" Jul 22 11:29:47.845840 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:47.845802 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n98hc" podStartSLOduration=2.301150196 podStartE2EDuration="4.845792809s" totalImagesPullingTime="2.544642613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:29:47.844942635 +0000 UTC m=+358.006485334" watchObservedRunningTime="2026-07-22 11:29:47.845792809 +0000 UTC m=+358.007335505" Jul 22 11:29:48.547110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:48.547072 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:48.650108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:48.650066 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:29:50.707080 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:29:50.707035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:06.599739 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:06.599695 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:06.694608 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:06.694571 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:07.299234 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.299197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:07.557797 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.557703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:07.660196 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.660162 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:07.694904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.694876 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m"] Jul 22 11:30:07.698378 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.698362 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.701238 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.701211 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 11:30:07.701341 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.701328 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 11:30:07.701443 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.701429 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 11:30:07.702159 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.702141 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 11:30:07.702296 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.702282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6rn5\"" Jul 22 11:30:07.713973 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.713926 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m"] Jul 22 11:30:07.859152 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.859089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.859152 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.859138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.859292 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.859213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-manifests\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.859292 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.859240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-tmp\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.859422 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.859289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56ds\" (UniqueName: \"kubernetes.io/projected/6e9df6e6-bcda-4214-b8d1-13fedd123f54-kube-api-access-t56ds\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.960557 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.960534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.960652 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.960576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.960652 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.960602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-manifests\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.960652 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.960618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-tmp\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.960652 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.960646 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56ds\" (UniqueName: \"kubernetes.io/projected/6e9df6e6-bcda-4214-b8d1-13fedd123f54-kube-api-access-t56ds\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.961025 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.961008 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-manifests\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.963092 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.963075 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e9df6e6-bcda-4214-b8d1-13fedd123f54-tmp\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.963178 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.963158 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.963235 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.963198 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e9df6e6-bcda-4214-b8d1-13fedd123f54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:07.972051 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:07.972024 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56ds\" (UniqueName: \"kubernetes.io/projected/6e9df6e6-bcda-4214-b8d1-13fedd123f54-kube-api-access-t56ds\") pod \"opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m\" (UID: \"6e9df6e6-bcda-4214-b8d1-13fedd123f54\") " pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:08.009981 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.009930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:08.151821 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.151791 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m"] Jul 22 11:30:08.154550 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:30:08.154527 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9df6e6_bcda_4214_b8d1_13fedd123f54.slice/crio-b4f7a439cba756e3b43cf5f822a8cf225bfe523e8a6e39c60e0a74824e397b80 WatchSource:0}: Error finding container b4f7a439cba756e3b43cf5f822a8cf225bfe523e8a6e39c60e0a74824e397b80: Status 404 returned error can't find the container with id b4f7a439cba756e3b43cf5f822a8cf225bfe523e8a6e39c60e0a74824e397b80 Jul 22 11:30:08.352403 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.352378 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:08.462254 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.462189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:08.554848 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.554825 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:08.904897 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:08.904862 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" event={"ID":"6e9df6e6-bcda-4214-b8d1-13fedd123f54","Type":"ContainerStarted","Data":"b4f7a439cba756e3b43cf5f822a8cf225bfe523e8a6e39c60e0a74824e397b80"} Jul 22 11:30:11.924826 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:11.924796 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" event={"ID":"6e9df6e6-bcda-4214-b8d1-13fedd123f54","Type":"ContainerStarted","Data":"ecd7a0eb0fb31670b7175cb416aa99b99a68409c66f8bb36ec094c59864e23f6"} Jul 22 11:30:12.930423 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:12.930389 2569 generic.go:356] "Generic (PLEG): container finished" podID="6e9df6e6-bcda-4214-b8d1-13fedd123f54" containerID="ecd7a0eb0fb31670b7175cb416aa99b99a68409c66f8bb36ec094c59864e23f6" exitCode=0 Jul 22 11:30:12.930932 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:12.930428 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" event={"ID":"6e9df6e6-bcda-4214-b8d1-13fedd123f54","Type":"ContainerDied","Data":"ecd7a0eb0fb31670b7175cb416aa99b99a68409c66f8bb36ec094c59864e23f6"} Jul 22 11:30:13.936451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:13.936419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" event={"ID":"6e9df6e6-bcda-4214-b8d1-13fedd123f54","Type":"ContainerStarted","Data":"2bcb7ebfd4e3d9930ff95e80299180ebaf8202beb248cafc33c17bf67fda2480"} Jul 22 11:30:13.936783 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:13.936566 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:13.960734 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:13.960689 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" podStartSLOduration=2.5950767900000002 podStartE2EDuration="6.960676956s" totalImagesPullingTime="3.293494959s" totalInitContainerRuntime="1.072105207s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:30:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:30:13.958495602 +0000 UTC m=+384.120038297" watchObservedRunningTime="2026-07-22 11:30:13.960676956 +0000 UTC m=+384.122219718" Jul 22 11:30:16.699149 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:16.699114 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:16.801543 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:16.801503 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:19.214174 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:19.214137 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:19.311602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:19.311575 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:24.942363 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:24.942337 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m" Jul 22 11:30:27.152648 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:27.152611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:27.271245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:27.271212 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:27.576904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:27.576853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:27.941746 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:27.941688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:28.232044 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.231989 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs"] Jul 22 11:30:28.240473 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.240457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.243548 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.243532 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z7lvc\"" Jul 22 11:30:28.243628 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.243550 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 11:30:28.244137 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.244121 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 11:30:28.244555 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.244541 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 11:30:28.245233 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.245218 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 11:30:28.257362 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.257341 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs"] Jul 22 11:30:28.304534 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.304513 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370974f3-afe9-436b-9bca-a7fd3c84cd02-tmp\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.304614 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.304555 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370974f3-afe9-436b-9bca-a7fd3c84cd02-tls-certs\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.304614 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.304592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc42v\" (UniqueName: \"kubernetes.io/projected/370974f3-afe9-436b-9bca-a7fd3c84cd02-kube-api-access-hc42v\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.405649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.405625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370974f3-afe9-436b-9bca-a7fd3c84cd02-tmp\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.405736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.405670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370974f3-afe9-436b-9bca-a7fd3c84cd02-tls-certs\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.405736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.405698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc42v\" (UniqueName: \"kubernetes.io/projected/370974f3-afe9-436b-9bca-a7fd3c84cd02-kube-api-access-hc42v\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.407793 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.407775 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/370974f3-afe9-436b-9bca-a7fd3c84cd02-tmp\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.408051 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.408036 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370974f3-afe9-436b-9bca-a7fd3c84cd02-tls-certs\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.435220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.435191 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc42v\" (UniqueName: \"kubernetes.io/projected/370974f3-afe9-436b-9bca-a7fd3c84cd02-kube-api-access-hc42v\") pod \"kube-auth-proxy-ffd4cd5bc-fjtgs\" (UID: \"370974f3-afe9-436b-9bca-a7fd3c84cd02\") " pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.549627 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.549610 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" Jul 22 11:30:28.660918 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.660822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:28.696503 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.696482 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs"] Jul 22 11:30:28.698129 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:30:28.698104 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370974f3_afe9_436b_9bca_a7fd3c84cd02.slice/crio-eeef364a52b031643dfa197eed8fadf8bf8da04341efb8ea218d05f30e42ec59 WatchSource:0}: Error finding container eeef364a52b031643dfa197eed8fadf8bf8da04341efb8ea218d05f30e42ec59: Status 404 returned error can't find the container with id eeef364a52b031643dfa197eed8fadf8bf8da04341efb8ea218d05f30e42ec59 Jul 22 11:30:28.791572 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.791543 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:28.921638 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.921574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:28.992425 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:28.992402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" event={"ID":"370974f3-afe9-436b-9bca-a7fd3c84cd02","Type":"ContainerStarted","Data":"eeef364a52b031643dfa197eed8fadf8bf8da04341efb8ea218d05f30e42ec59"} Jul 22 11:30:29.027720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:29.027698 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:29.796805 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:29.796771 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:30.503076 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:30.503034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:30.656039 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:30.655992 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:31.002411 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:31.002376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" event={"ID":"370974f3-afe9-436b-9bca-a7fd3c84cd02","Type":"ContainerStarted","Data":"19ebbd62f78408cd9954b1a9a5f07824c6240d619f0cf2222ed0e663e7b2d804"} Jul 22 11:30:31.036909 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:31.036861 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-ffd4cd5bc-fjtgs" podStartSLOduration=0.864440507 podStartE2EDuration="3.036838839s" totalImagesPullingTime="2.172398332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:30:31.034639138 +0000 UTC m=+401.196181825" watchObservedRunningTime="2026-07-22 11:30:31.036838839 +0000 UTC m=+401.198381579" Jul 22 11:30:32.250694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:32.250657 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:36.099733 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.099690 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:36.227711 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.227687 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:36.974431 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.974404 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv"] Jul 22 11:30:36.976588 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.976574 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:36.978709 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.978687 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 11:30:36.978831 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.978687 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r8qjj\"" Jul 22 11:30:36.993986 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:36.993940 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv"] Jul 22 11:30:37.000560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.000537 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:37.053159 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053250 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053250 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053250 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e859e1-8863-422c-9a49-d8749e5b334f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053250 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053375 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtn8\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-kube-api-access-txtn8\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053375 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053436 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053375 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.053436 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.053394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e859e1-8863-422c-9a49-d8749e5b334f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.153732 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e859e1-8863-422c-9a49-d8749e5b334f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e859e1-8863-422c-9a49-d8749e5b334f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153827 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtn8\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-kube-api-access-txtn8\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154090 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.153940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154525 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.154143 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154525 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.154193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154525 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.154420 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154618 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.154524 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.154618 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.154580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e859e1-8863-422c-9a49-d8749e5b334f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.156120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.156103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e859e1-8863-422c-9a49-d8749e5b334f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.156197 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.156180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e859e1-8863-422c-9a49-d8749e5b334f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.169232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.169210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.172639 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.172620 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtn8\" (UniqueName: \"kubernetes.io/projected/90e859e1-8863-422c-9a49-d8749e5b334f-kube-api-access-txtn8\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dbmnmv\" (UID: \"90e859e1-8863-422c-9a49-d8749e5b334f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.208973 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.208923 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7"] Jul 22 11:30:37.211649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.211631 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.217226 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.217203 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 11:30:37.217341 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.217263 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sfgp6\"" Jul 22 11:30:37.218694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.218676 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 11:30:37.219454 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.219432 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 11:30:37.239583 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.239537 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7"] Jul 22 11:30:37.254302 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68288dc1-7b9d-49d8-82e4-20b7d246e510-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnx52\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-kube-api-access-gnx52\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254415 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.254478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.254442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.286458 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.286438 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:37.306776 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.306751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:37.354943 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.354868 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.354943 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.354907 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.354948 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68288dc1-7b9d-49d8-82e4-20b7d246e510-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.355042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnx52\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-kube-api-access-gnx52\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355348 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.355301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.355431 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.355441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.355585 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.355497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.357708 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.357671 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.358021 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.357945 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68288dc1-7b9d-49d8-82e4-20b7d246e510-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.358082 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.358051 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.358495 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.358479 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.385741 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.385681 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.392648 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.392626 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnx52\" (UniqueName: \"kubernetes.io/projected/68288dc1-7b9d-49d8-82e4-20b7d246e510-kube-api-access-gnx52\") pod \"istiod-openshift-gateway-94bb8fbfd-4gdj7\" (UID: \"68288dc1-7b9d-49d8-82e4-20b7d246e510\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.428766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.428742 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:37.448144 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.448121 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv"] Jul 22 11:30:37.451165 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:30:37.451144 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e859e1_8863_422c_9a49_d8749e5b334f.slice/crio-abdc6fa7fb4ede3f1c9ee3a02a3964bab85dea40d09e646188d13c6e8cd0bc55 WatchSource:0}: Error finding container abdc6fa7fb4ede3f1c9ee3a02a3964bab85dea40d09e646188d13c6e8cd0bc55: Status 404 returned error can't find the container with id abdc6fa7fb4ede3f1c9ee3a02a3964bab85dea40d09e646188d13c6e8cd0bc55 Jul 22 11:30:37.520814 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.520767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:37.656391 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:37.656367 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7"] Jul 22 11:30:37.658573 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:30:37.658547 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68288dc1_7b9d_49d8_82e4_20b7d246e510.slice/crio-dc6d9564cfba091b2e5850fd96b901a37899d48e9f7de7b5c5921bcad7cbe308 WatchSource:0}: Error finding container dc6d9564cfba091b2e5850fd96b901a37899d48e9f7de7b5c5921bcad7cbe308: Status 404 returned error can't find the container with id dc6d9564cfba091b2e5850fd96b901a37899d48e9f7de7b5c5921bcad7cbe308 Jul 22 11:30:38.028085 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:38.028060 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" event={"ID":"90e859e1-8863-422c-9a49-d8749e5b334f","Type":"ContainerStarted","Data":"abdc6fa7fb4ede3f1c9ee3a02a3964bab85dea40d09e646188d13c6e8cd0bc55"} Jul 22 11:30:38.029249 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:38.029226 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" event={"ID":"68288dc1-7b9d-49d8-82e4-20b7d246e510","Type":"ContainerStarted","Data":"dc6d9564cfba091b2e5850fd96b901a37899d48e9f7de7b5c5921bcad7cbe308"} Jul 22 11:30:40.555287 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:40.555250 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:40.666117 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:40.666083 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:40.862599 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:40.862515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:40.970569 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:40.970523 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:41.079991 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.079761 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:41.243330 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.243256 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:30:41.743730 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.743695 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:30:41.744024 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.743762 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:30:41.744024 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.743755 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:30:41.744024 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.743888 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:30:41.744024 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:41.743916 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:30:42.054268 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.054234 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" event={"ID":"90e859e1-8863-422c-9a49-d8749e5b334f","Type":"ContainerStarted","Data":"bdfff0b1be9c286f0a7c4fbf1bbfb9ab1b0cdea5bdb9ddc1d710188daff34f91"} Jul 22 11:30:42.055942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.055912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" event={"ID":"68288dc1-7b9d-49d8-82e4-20b7d246e510","Type":"ContainerStarted","Data":"e41a7ab89331e5d8d8368c49682efae0b34a5c8ac0bbdc6e8d379ea4c6792cce"} Jul 22 11:30:42.056106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.056090 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:42.057537 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.057516 2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-4gdj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 11:30:42.057640 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.057566 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" podUID="68288dc1-7b9d-49d8-82e4-20b7d246e510" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 11:30:42.117574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.117392 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" podStartSLOduration=1.8267862780000002 podStartE2EDuration="6.117377547s" totalImagesPullingTime="4.290591269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:30:42.116264892 +0000 UTC m=+412.277807587" watchObservedRunningTime="2026-07-22 11:30:42.117377547 +0000 UTC m=+412.278920244" Jul 22 11:30:42.194727 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.194692 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" podStartSLOduration=1.111655599 podStartE2EDuration="5.194680825s" totalImagesPullingTime="4.083025226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:30:42.194050765 +0000 UTC m=+412.355593470" watchObservedRunningTime="2026-07-22 11:30:42.194680825 +0000 UTC m=+412.356223521" Jul 22 11:30:42.287376 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.287354 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:42.288774 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.288750 2569 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 22 11:30:42.288834 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:42.288796 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" podUID="90e859e1-8863-422c-9a49-d8749e5b334f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 22 11:30:43.060684 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:43.060651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gdj7" Jul 22 11:30:43.289853 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:43.289826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:44.064596 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:44.064563 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:44.065572 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:44.065552 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dbmnmv" Jul 22 11:30:58.573428 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:30:58.573396 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:24.675188 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:24.675101 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:28.832162 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:28.832127 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:28.935193 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:28.935152 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:29.041706 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:29.041674 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:52.053467 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:52.053430 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:52.194248 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:52.194206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:52.302488 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:52.302456 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:52.628935 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:52.628884 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:53.067937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:53.067906 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:53.259625 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:53.259595 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:57.055356 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.055324 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:57.350536 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.350457 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b68c87c5f-ghnw4"] Jul 22 11:31:57.352726 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.352705 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370211 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gsl\" (UniqueName: \"kubernetes.io/projected/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-kube-api-access-m7gsl\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-service-ca\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370285 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-oauth-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370339 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-trusted-ca-bundle\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.370515 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.370358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-oauth-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.398205 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.398174 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b68c87c5f-ghnw4"] Jul 22 11:31:57.470756 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.470756 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gsl\" (UniqueName: \"kubernetes.io/projected/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-kube-api-access-m7gsl\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470779 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-service-ca\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-oauth-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-trusted-ca-bundle\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.470910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-oauth-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471706 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.471646 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471706 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.471670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-service-ca\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.471706 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.471671 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-oauth-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.472097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.472071 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-trusted-ca-bundle\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.473444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.473424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-oauth-config\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.473560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.473544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-console-serving-cert\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.481808 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.481786 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gsl\" (UniqueName: \"kubernetes.io/projected/fb2ed0c0-76bd-4f92-8093-89d95d42c03b-kube-api-access-m7gsl\") pod \"console-5b68c87c5f-ghnw4\" (UID: \"fb2ed0c0-76bd-4f92-8093-89d95d42c03b\") " pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.494102 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.494072 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:57.603821 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.603736 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:57.661814 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.661779 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:31:57.705052 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.705019 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:57.801327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:57.801299 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b68c87c5f-ghnw4"] Jul 22 11:31:57.803208 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:31:57.803179 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2ed0c0_76bd_4f92_8093_89d95d42c03b.slice/crio-d4aca808640910605ee45e761aa3931c6ef72fd28ddef2467044e8a9c7ac000b WatchSource:0}: Error finding container d4aca808640910605ee45e761aa3931c6ef72fd28ddef2467044e8a9c7ac000b: Status 404 returned error can't find the container with id d4aca808640910605ee45e761aa3931c6ef72fd28ddef2467044e8a9c7ac000b Jul 22 11:31:58.198058 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:58.197947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:58.369893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:58.369861 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b68c87c5f-ghnw4" event={"ID":"fb2ed0c0-76bd-4f92-8093-89d95d42c03b","Type":"ContainerStarted","Data":"ad8d0986c76ef341b7838fc90848e0eccdecec0ce39ea4134b0d0c41b9ee5742"} Jul 22 11:31:58.370061 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:58.369914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b68c87c5f-ghnw4" event={"ID":"fb2ed0c0-76bd-4f92-8093-89d95d42c03b","Type":"ContainerStarted","Data":"d4aca808640910605ee45e761aa3931c6ef72fd28ddef2467044e8a9c7ac000b"} Jul 22 11:31:58.397148 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:58.397116 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:31:58.403453 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:58.403414 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b68c87c5f-ghnw4" podStartSLOduration=1.403400435 podStartE2EDuration="1.403400435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:31:58.402027796 +0000 UTC m=+488.563570491" watchObservedRunningTime="2026-07-22 11:31:58.403400435 +0000 UTC m=+488.564943130" Jul 22 11:31:59.387256 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.387223 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8"] Jul 22 11:31:59.389414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.389396 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:31:59.394570 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.394541 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 11:31:59.395173 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.395154 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7p25m\"" Jul 22 11:31:59.395267 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.395170 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 11:31:59.417717 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.417683 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8"] Jul 22 11:31:59.492213 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.492173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kjc\" (UniqueName: \"kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc\") pod \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" (UID: \"7a603b98-115b-4711-b892-e6ae7611d222\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:31:59.593074 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.593035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kjc\" (UniqueName: \"kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc\") pod \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" (UID: \"7a603b98-115b-4711-b892-e6ae7611d222\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:31:59.607601 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.607568 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kjc\" (UniqueName: \"kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc\") pod \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" (UID: \"7a603b98-115b-4711-b892-e6ae7611d222\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:31:59.700117 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.700027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:31:59.840565 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:31:59.840543 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8"] Jul 22 11:31:59.843226 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:31:59.843197 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a603b98_115b_4711_b892_e6ae7611d222.slice/crio-aa70d006483c3b64d529ef6d254e9b58bd72b7822a0ec9f6668c07832be5e0ae WatchSource:0}: Error finding container aa70d006483c3b64d529ef6d254e9b58bd72b7822a0ec9f6668c07832be5e0ae: Status 404 returned error can't find the container with id aa70d006483c3b64d529ef6d254e9b58bd72b7822a0ec9f6668c07832be5e0ae Jul 22 11:32:00.379318 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:00.379280 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" event={"ID":"7a603b98-115b-4711-b892-e6ae7611d222","Type":"ContainerStarted","Data":"aa70d006483c3b64d529ef6d254e9b58bd72b7822a0ec9f6668c07832be5e0ae"} Jul 22 11:32:00.655028 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:00.654930 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:01.087659 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:01.087546 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:02.390095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:02.390062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" event={"ID":"7a603b98-115b-4711-b892-e6ae7611d222","Type":"ContainerStarted","Data":"e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a"} Jul 22 11:32:02.390510 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:02.390168 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:02.422767 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:02.422729 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" podStartSLOduration=1.618204124 podStartE2EDuration="3.422713828s" totalImagesPullingTime="1.804509704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:32:02.419832233 +0000 UTC m=+492.581374942" watchObservedRunningTime="2026-07-22 11:32:02.422713828 +0000 UTC m=+492.584256572" Jul 22 11:32:04.065159 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:04.065116 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:04.326780 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:04.326694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:04.475755 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:04.475720 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:04.595019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:04.594908 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:07.662482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:07.662447 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:32:07.662924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:07.662596 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:32:07.667327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:07.667307 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:32:08.418564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:08.418540 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b68c87c5f-ghnw4" Jul 22 11:32:08.477597 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:08.477567 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:32:08.598109 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:08.598076 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:08.698034 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:08.697923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:11.066035 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:11.065997 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:11.172613 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:11.172562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:11.685098 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:11.685068 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:11.779782 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:11.779746 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:13.396512 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:13.396484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:15.435352 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:15.435313 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:15.537908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:15.537872 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:17.167601 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:17.167562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:17.505293 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:17.505207 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:18.503086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:18.503052 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:18.613711 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:18.613675 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:25.688577 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:25.688536 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:27.116436 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.116388 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc"] Jul 22 11:32:27.123527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.123488 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.125922 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.125900 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5chdv\"" Jul 22 11:32:27.132246 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.132222 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc"] Jul 22 11:32:27.216807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.216780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsmg\" (UniqueName: \"kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.216934 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.216845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.317701 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.317670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.317809 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.317739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsmg\" (UniqueName: \"kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.318071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.318052 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.328132 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.328113 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsmg\" (UniqueName: \"kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.435372 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.435322 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:27.577426 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.577345 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc"] Jul 22 11:32:27.581384 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:32:27.581358 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f0ad12_fe5b_4c75_a502_901e3114437c.slice/crio-b33bf81556c1d13681643742db1ea520e8a37c203f59dedd543fb6f9c3f473f6 WatchSource:0}: Error finding container b33bf81556c1d13681643742db1ea520e8a37c203f59dedd543fb6f9c3f473f6: Status 404 returned error can't find the container with id b33bf81556c1d13681643742db1ea520e8a37c203f59dedd543fb6f9c3f473f6 Jul 22 11:32:27.919238 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.919203 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc"] Jul 22 11:32:27.937794 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.937768 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc"] Jul 22 11:32:27.978700 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.978670 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:27.983392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:27.983376 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.005643 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.005618 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8"] Jul 22 11:32:28.005832 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.005790 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" podUID="7a603b98-115b-4711-b892-e6ae7611d222" containerName="manager" containerID="cri-o://e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a" gracePeriod=2 Jul 22 11:32:28.013094 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.013070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:28.022350 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.022324 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8"] Jul 22 11:32:28.050275 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:28.050239 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="7a603b98-115b-4711-b892-e6ae7611d222" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:28.122971 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.122921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrk5r\" (UniqueName: \"kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.123449 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.123023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.129220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.129192 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp"] Jul 22 11:32:28.129667 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.129630 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a603b98-115b-4711-b892-e6ae7611d222" containerName="manager" Jul 22 11:32:28.129667 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.129651 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a603b98-115b-4711-b892-e6ae7611d222" containerName="manager" Jul 22 11:32:28.129797 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.129734 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a603b98-115b-4711-b892-e6ae7611d222" containerName="manager" Jul 22 11:32:28.132728 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.132707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:28.149772 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.149744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp"] Jul 22 11:32:28.224693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.224643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrk5r\" (UniqueName: \"kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.224693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.224702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5pp\" (UniqueName: \"kubernetes.io/projected/9165cfd9-1d39-455e-8cf0-e9c858320342-kube-api-access-qd5pp\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zlp\" (UID: \"9165cfd9-1d39-455e-8cf0-e9c858320342\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:28.224942 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.224760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.225299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.225207 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.241330 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.241201 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrk5r\" (UniqueName: \"kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jkc6q\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.263081 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:28.263050 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="7a603b98-115b-4711-b892-e6ae7611d222" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:28.293911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.293891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:28.295048 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.295032 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:28.305765 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:28.305733 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6c6w8\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="7a603b98-115b-4711-b892-e6ae7611d222" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:28.326262 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.326233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5pp\" (UniqueName: \"kubernetes.io/projected/9165cfd9-1d39-455e-8cf0-e9c858320342-kube-api-access-qd5pp\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zlp\" (UID: \"9165cfd9-1d39-455e-8cf0-e9c858320342\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:28.356940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.355241 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5pp\" (UniqueName: \"kubernetes.io/projected/9165cfd9-1d39-455e-8cf0-e9c858320342-kube-api-access-qd5pp\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zlp\" (UID: \"9165cfd9-1d39-455e-8cf0-e9c858320342\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:28.426945 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.426902 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54kjc\" (UniqueName: \"kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc\") pod \"7a603b98-115b-4711-b892-e6ae7611d222\" (UID: \"7a603b98-115b-4711-b892-e6ae7611d222\") " Jul 22 11:32:28.429975 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.429918 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc" (OuterVolumeSpecName: "kube-api-access-54kjc") pod "7a603b98-115b-4711-b892-e6ae7611d222" (UID: "7a603b98-115b-4711-b892-e6ae7611d222"). InnerVolumeSpecName "kube-api-access-54kjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:32:28.445066 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.445002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:28.501823 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.501788 2569 generic.go:356] "Generic (PLEG): container finished" podID="7a603b98-115b-4711-b892-e6ae7611d222" containerID="e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a" exitCode=0 Jul 22 11:32:28.502034 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.501837 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6c6w8" Jul 22 11:32:28.502034 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.501864 2569 scope.go:117] "RemoveContainer" containerID="e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a" Jul 22 11:32:28.528735 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.528700 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54kjc\" (UniqueName: \"kubernetes.io/projected/7a603b98-115b-4711-b892-e6ae7611d222-kube-api-access-54kjc\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:28.561326 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.561275 2569 scope.go:117] "RemoveContainer" containerID="e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a" Jul 22 11:32:28.562764 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:28.562739 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a\": container with ID starting with e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a not found: ID does not exist" containerID="e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a" Jul 22 11:32:28.562893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.562770 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a"} err="failed to get container status \"e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a\": rpc error: code = NotFound desc = could not find container \"e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a\": container with ID starting with e8a86821a6d011fb757b00180efc669a09f9ddae2bef20c1ba037df78099829a not found: ID does not exist" Jul 22 11:32:28.654891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.654849 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:28.669094 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:32:28.669062 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7943241_daa6_46c4_9b26_20e85b772f3c.slice/crio-01110d1b4ebbf6d43daedf770b4a97a2d05ef28863a3e885b7e00fd0d6e783ce WatchSource:0}: Error finding container 01110d1b4ebbf6d43daedf770b4a97a2d05ef28863a3e885b7e00fd0d6e783ce: Status 404 returned error can't find the container with id 01110d1b4ebbf6d43daedf770b4a97a2d05ef28863a3e885b7e00fd0d6e783ce Jul 22 11:32:28.782509 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:28.782461 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp"] Jul 22 11:32:28.785535 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:32:28.785498 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9165cfd9_1d39_455e_8cf0_e9c858320342.slice/crio-40622beb1517fa11f4088681c97b40ecb12435b6d851775500a435eaa1f72e82 WatchSource:0}: Error finding container 40622beb1517fa11f4088681c97b40ecb12435b6d851775500a435eaa1f72e82: Status 404 returned error can't find the container with id 40622beb1517fa11f4088681c97b40ecb12435b6d851775500a435eaa1f72e82 Jul 22 11:32:29.509401 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:29.509359 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" event={"ID":"e7943241-daa6-46c4-9b26-20e85b772f3c","Type":"ContainerStarted","Data":"01110d1b4ebbf6d43daedf770b4a97a2d05ef28863a3e885b7e00fd0d6e783ce"} Jul 22 11:32:29.511783 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:29.511448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" event={"ID":"9165cfd9-1d39-455e-8cf0-e9c858320342","Type":"ContainerStarted","Data":"ff6e30fc8652b7379cd677eca5a5d3532acd813000e342a908d45ad469d6528e"} Jul 22 11:32:29.511783 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:29.511499 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" event={"ID":"9165cfd9-1d39-455e-8cf0-e9c858320342","Type":"ContainerStarted","Data":"40622beb1517fa11f4088681c97b40ecb12435b6d851775500a435eaa1f72e82"} Jul 22 11:32:29.511783 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:29.511551 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:29.533409 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:29.533359 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" podStartSLOduration=1.533342454 podStartE2EDuration="1.533342454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:32:29.532010743 +0000 UTC m=+519.693553467" watchObservedRunningTime="2026-07-22 11:32:29.533342454 +0000 UTC m=+519.694885153" Jul 22 11:32:30.351396 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:30.351360 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a603b98-115b-4711-b892-e6ae7611d222" path="/var/lib/kubelet/pods/7a603b98-115b-4711-b892-e6ae7611d222/volumes" Jul 22 11:32:31.458984 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:31.458902 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:31.558634 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:31.558592 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:31.878696 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:31.878653 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:32.032327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.032242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:32.530752 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.530719 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" event={"ID":"e7943241-daa6-46c4-9b26-20e85b772f3c","Type":"ContainerStarted","Data":"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584"} Jul 22 11:32:32.531196 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.530798 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:32.532450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.532417 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" containerName="manager" containerID="cri-o://f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b" gracePeriod=2 Jul 22 11:32:32.561243 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:32.558903 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:32.561243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.559661 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" podStartSLOduration=2.5434499 podStartE2EDuration="5.559635626s" totalImagesPullingTime="3.016185726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:32:32.557207062 +0000 UTC m=+522.718749778" watchObservedRunningTime="2026-07-22 11:32:32.559635626 +0000 UTC m=+522.721178323" Jul 22 11:32:32.765902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.765881 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:32.767997 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:32.767941 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:32.866896 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.866868 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume\") pod \"45f0ad12-fe5b-4c75-a502-901e3114437c\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " Jul 22 11:32:32.867049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.866938 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gsmg\" (UniqueName: \"kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg\") pod \"45f0ad12-fe5b-4c75-a502-901e3114437c\" (UID: \"45f0ad12-fe5b-4c75-a502-901e3114437c\") " Jul 22 11:32:32.867145 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.867118 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "45f0ad12-fe5b-4c75-a502-901e3114437c" (UID: "45f0ad12-fe5b-4c75-a502-901e3114437c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:32:32.869110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.869088 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg" (OuterVolumeSpecName: "kube-api-access-5gsmg") pod "45f0ad12-fe5b-4c75-a502-901e3114437c" (UID: "45f0ad12-fe5b-4c75-a502-901e3114437c"). InnerVolumeSpecName "kube-api-access-5gsmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:32:32.967627 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.967585 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0ad12-fe5b-4c75-a502-901e3114437c-extensions-socket-volume\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:32.967627 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:32.967624 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsmg\" (UniqueName: \"kubernetes.io/projected/45f0ad12-fe5b-4c75-a502-901e3114437c-kube-api-access-5gsmg\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.498490 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.498445 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b5795dcf7-gl2xt" podUID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" containerName="console" containerID="cri-o://f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514" gracePeriod=15 Jul 22 11:32:33.537885 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.537856 2569 generic.go:356] "Generic (PLEG): container finished" podID="45f0ad12-fe5b-4c75-a502-901e3114437c" containerID="f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b" exitCode=2 Jul 22 11:32:33.538210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.537905 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:33.538210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.537967 2569 scope.go:117] "RemoveContainer" containerID="f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b" Jul 22 11:32:33.539865 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:33.539835 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:33.569077 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:33.569051 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:33.610240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.610213 2569 scope.go:117] "RemoveContainer" containerID="f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b" Jul 22 11:32:33.610551 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:33.610531 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b\": container with ID starting with f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b not found: ID does not exist" containerID="f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b" Jul 22 11:32:33.610606 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.610559 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b"} err="failed to get container status \"f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b\": rpc error: code = NotFound desc = could not find container \"f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b\": container with ID starting with f6f7c7c2bc2b87151fe48560d4a015de7a5185b271b29420cdc760d60126374b not found: ID does not exist" Jul 22 11:32:33.744101 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.744081 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:32:33.771899 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:33.771837 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhncc\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhncc" Jul 22 11:32:33.875094 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875067 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875116 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875150 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875167 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875187 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875304 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlfgj\" (UniqueName: \"kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875363 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert\") pod \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\" (UID: \"6abb9afd-85f3-492c-92c4-34f5ecdc85e4\") " Jul 22 11:32:33.875627 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875589 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:32:33.875742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875628 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config" (OuterVolumeSpecName: "console-config") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:32:33.875742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875659 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:32:33.875742 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.875684 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:32:33.877564 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.877533 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj" (OuterVolumeSpecName: "kube-api-access-rlfgj") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "kube-api-access-rlfgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:32:33.877673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.877615 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:32:33.877751 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.877730 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6abb9afd-85f3-492c-92c4-34f5ecdc85e4" (UID: "6abb9afd-85f3-492c-92c4-34f5ecdc85e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:32:33.976289 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976266 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-oauth-config\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976289 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976286 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-config\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976294 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976302 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-oauth-serving-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976310 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfgj\" (UniqueName: \"kubernetes.io/projected/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-kube-api-access-rlfgj\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976318 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-console-serving-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:33.976415 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:33.976326 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abb9afd-85f3-492c-92c4-34f5ecdc85e4-trusted-ca-bundle\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:34.351000 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.350940 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" path="/var/lib/kubelet/pods/45f0ad12-fe5b-4c75-a502-901e3114437c/volumes" Jul 22 11:32:34.543412 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.543379 2569 generic.go:356] "Generic (PLEG): container finished" podID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" containerID="f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514" exitCode=0 Jul 22 11:32:34.543856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.543427 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5795dcf7-gl2xt" Jul 22 11:32:34.543856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.543463 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5795dcf7-gl2xt" event={"ID":"6abb9afd-85f3-492c-92c4-34f5ecdc85e4","Type":"ContainerDied","Data":"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514"} Jul 22 11:32:34.543856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.543509 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5795dcf7-gl2xt" event={"ID":"6abb9afd-85f3-492c-92c4-34f5ecdc85e4","Type":"ContainerDied","Data":"2963be56b97f5af1a141f69652952b2c2dd84b9fc407a6481ed26852ff5711e2"} Jul 22 11:32:34.543856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.543530 2569 scope.go:117] "RemoveContainer" containerID="f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514" Jul 22 11:32:34.553907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.553886 2569 scope.go:117] "RemoveContainer" containerID="f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514" Jul 22 11:32:34.554226 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:34.554204 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514\": container with ID starting with f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514 not found: ID does not exist" containerID="f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514" Jul 22 11:32:34.554299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.554236 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514"} err="failed to get container status \"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514\": rpc error: code = NotFound desc = could not find container \"f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514\": container with ID starting with f4162463354b523988a4c798a52ec3bed54aa577b0f3509f6e72b5ed20f0a514 not found: ID does not exist" Jul 22 11:32:34.566790 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.566758 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:32:34.569444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:34.569419 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5795dcf7-gl2xt"] Jul 22 11:32:35.434697 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:35.434664 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:35.888008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:35.887944 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:36.349693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:36.349651 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" path="/var/lib/kubelet/pods/6abb9afd-85f3-492c-92c4-34f5ecdc85e4/volumes" Jul 22 11:32:38.836998 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:38.836945 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:38.937554 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:38.937521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:40.521601 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:40.521523 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zlp" Jul 22 11:32:43.056033 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:43.056000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:43.155867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:43.155832 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:43.540776 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:43.540701 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:48.494631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:48.494593 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:32:56.311750 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.311712 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:56.312170 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.312013 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" podUID="e7943241-daa6-46c4-9b26-20e85b772f3c" containerName="manager" containerID="cri-o://f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584" gracePeriod=10 Jul 22 11:32:56.556786 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.556762 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:56.644468 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.644388 2569 generic.go:356] "Generic (PLEG): container finished" podID="e7943241-daa6-46c4-9b26-20e85b772f3c" containerID="f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584" exitCode=0 Jul 22 11:32:56.644468 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.644437 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" Jul 22 11:32:56.644647 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.644481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" event={"ID":"e7943241-daa6-46c4-9b26-20e85b772f3c","Type":"ContainerDied","Data":"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584"} Jul 22 11:32:56.644647 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.644537 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q" event={"ID":"e7943241-daa6-46c4-9b26-20e85b772f3c","Type":"ContainerDied","Data":"01110d1b4ebbf6d43daedf770b4a97a2d05ef28863a3e885b7e00fd0d6e783ce"} Jul 22 11:32:56.644647 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.644563 2569 scope.go:117] "RemoveContainer" containerID="f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584" Jul 22 11:32:56.649711 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.649693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrk5r\" (UniqueName: \"kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r\") pod \"e7943241-daa6-46c4-9b26-20e85b772f3c\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " Jul 22 11:32:56.649802 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.649792 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume\") pod \"e7943241-daa6-46c4-9b26-20e85b772f3c\" (UID: \"e7943241-daa6-46c4-9b26-20e85b772f3c\") " Jul 22 11:32:56.650188 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.650165 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7943241-daa6-46c4-9b26-20e85b772f3c" (UID: "e7943241-daa6-46c4-9b26-20e85b772f3c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:32:56.651743 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.651724 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r" (OuterVolumeSpecName: "kube-api-access-nrk5r") pod "e7943241-daa6-46c4-9b26-20e85b772f3c" (UID: "e7943241-daa6-46c4-9b26-20e85b772f3c"). InnerVolumeSpecName "kube-api-access-nrk5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:32:56.653315 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.653290 2569 scope.go:117] "RemoveContainer" containerID="f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584" Jul 22 11:32:56.653566 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:32:56.653548 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584\": container with ID starting with f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584 not found: ID does not exist" containerID="f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584" Jul 22 11:32:56.653630 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.653579 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584"} err="failed to get container status \"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584\": rpc error: code = NotFound desc = could not find container \"f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584\": container with ID starting with f61f99c3f3b2f5b19c40fd6215eb6a7e5ab7926a5a98c1a315812f28782d7584 not found: ID does not exist" Jul 22 11:32:56.751310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.751279 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrk5r\" (UniqueName: \"kubernetes.io/projected/e7943241-daa6-46c4-9b26-20e85b772f3c-kube-api-access-nrk5r\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:56.751310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.751306 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7943241-daa6-46c4-9b26-20e85b772f3c-extensions-socket-volume\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:32:56.976975 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.976931 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:56.982767 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:56.982747 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jkc6q"] Jul 22 11:32:58.349053 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:32:58.349012 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7943241-daa6-46c4-9b26-20e85b772f3c" path="/var/lib/kubelet/pods/e7943241-daa6-46c4-9b26-20e85b772f3c/volumes" Jul 22 11:33:00.177856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.177825 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6"] Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178263 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" containerName="console" Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178282 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" containerName="console" Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178310 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" containerName="manager" Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178318 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" containerName="manager" Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178339 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7943241-daa6-46c4-9b26-20e85b772f3c" containerName="manager" Jul 22 11:33:00.178407 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178348 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7943241-daa6-46c4-9b26-20e85b772f3c" containerName="manager" Jul 22 11:33:00.178693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178423 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7943241-daa6-46c4-9b26-20e85b772f3c" containerName="manager" Jul 22 11:33:00.178693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178435 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abb9afd-85f3-492c-92c4-34f5ecdc85e4" containerName="console" Jul 22 11:33:00.178693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.178445 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f0ad12-fe5b-4c75-a502-901e3114437c" containerName="manager" Jul 22 11:33:00.182688 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.182668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.184928 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.184909 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tvs2v\"" Jul 22 11:33:00.193993 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.193948 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6"] Jul 22 11:33:00.283318 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx9b\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-kube-api-access-8nx9b\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283439 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283439 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283476 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d412bde1-a4b9-4024-9815-879a50c84655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283663 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283663 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.283663 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.283601 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d412bde1-a4b9-4024-9815-879a50c84655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.383976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.383931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.383984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d412bde1-a4b9-4024-9815-879a50c84655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384075 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d412bde1-a4b9-4024-9815-879a50c84655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx9b\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-kube-api-access-8nx9b\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384342 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384431 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384582 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384712 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.384947 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.384899 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d412bde1-a4b9-4024-9815-879a50c84655-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.386499 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.386480 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d412bde1-a4b9-4024-9815-879a50c84655-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.386616 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.386597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d412bde1-a4b9-4024-9815-879a50c84655-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.393170 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.393149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.393472 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.393448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx9b\" (UniqueName: \"kubernetes.io/projected/d412bde1-a4b9-4024-9815-879a50c84655-kube-api-access-8nx9b\") pod \"maas-default-gateway-openshift-default-5df9b4f89-zl9g6\" (UID: \"d412bde1-a4b9-4024-9815-879a50c84655\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.493475 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.493421 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:00.575697 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.575663 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:00.632691 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.632663 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6"] Jul 22 11:33:00.636261 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:33:00.636229 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd412bde1_a4b9_4024_9815_879a50c84655.slice/crio-0424da4c3989a9d4e840b8d000d49b2869279d6d57cccd4bd7053c9d1e5eab30 WatchSource:0}: Error finding container 0424da4c3989a9d4e840b8d000d49b2869279d6d57cccd4bd7053c9d1e5eab30: Status 404 returned error can't find the container with id 0424da4c3989a9d4e840b8d000d49b2869279d6d57cccd4bd7053c9d1e5eab30 Jul 22 11:33:00.638349 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.638322 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:33:00.638428 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.638383 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:33:00.638428 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.638412 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:33:00.663788 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.663765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" event={"ID":"d412bde1-a4b9-4024-9815-879a50c84655","Type":"ContainerStarted","Data":"0424da4c3989a9d4e840b8d000d49b2869279d6d57cccd4bd7053c9d1e5eab30"} Jul 22 11:33:00.684974 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:00.684932 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:01.670187 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:01.670139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" event={"ID":"d412bde1-a4b9-4024-9815-879a50c84655","Type":"ContainerStarted","Data":"162b421d181f8317a4720703708e858989af38d26c9c2df63bbaff8ddabb0736"} Jul 22 11:33:01.691645 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:01.691589 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" podStartSLOduration=1.691571799 podStartE2EDuration="1.691571799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:33:01.690251596 +0000 UTC m=+551.851794293" watchObservedRunningTime="2026-07-22 11:33:01.691571799 +0000 UTC m=+551.853114497" Jul 22 11:33:02.493875 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:02.493840 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:02.498545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:02.498523 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:02.673918 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:02.673883 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:02.674738 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:02.674720 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:33:07.773334 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:07.773301 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:14.701316 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.701277 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:14.704424 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.704403 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.706813 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.706793 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 11:33:14.706908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.706811 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hc5sn\"" Jul 22 11:33:14.712018 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.711997 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:14.716049 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.716029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:14.780766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.780736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cwv\" (UniqueName: \"kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.780766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.780768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.800439 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.800405 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:14.881927 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.881894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cwv\" (UniqueName: \"kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.882084 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.881929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.882571 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.882554 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:14.893327 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:14.893306 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cwv\" (UniqueName: \"kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv\") pod \"limitador-limitador-64c94fd64f-x7tv2\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:15.015556 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.015499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:15.143743 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.143721 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:15.146324 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:33:15.146300 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801c5378_127c_49f6_8bea_562ca9ccb4c1.slice/crio-2a2462c7735792e57fcfe8da489382497767613fe6b46d02fc9a90eed248dad3 WatchSource:0}: Error finding container 2a2462c7735792e57fcfe8da489382497767613fe6b46d02fc9a90eed248dad3: Status 404 returned error can't find the container with id 2a2462c7735792e57fcfe8da489382497767613fe6b46d02fc9a90eed248dad3 Jul 22 11:33:15.474278 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.474251 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:15.606216 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.606193 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:15.677406 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.677373 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:15.737807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:15.737745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" event={"ID":"801c5378-127c-49f6-8bea-562ca9ccb4c1","Type":"ContainerStarted","Data":"2a2462c7735792e57fcfe8da489382497767613fe6b46d02fc9a90eed248dad3"} Jul 22 11:33:18.756638 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:18.756604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" event={"ID":"801c5378-127c-49f6-8bea-562ca9ccb4c1","Type":"ContainerStarted","Data":"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e"} Jul 22 11:33:18.757045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:18.756739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:18.774598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:18.774553 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" podStartSLOduration=2.216572966 podStartE2EDuration="4.774541385s" totalImagesPullingTime="2.557968419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:33:18.773334642 +0000 UTC m=+568.934877339" watchObservedRunningTime="2026-07-22 11:33:18.774541385 +0000 UTC m=+568.936084079" Jul 22 11:33:29.762336 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:29.762306 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:30.495970 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:30.495932 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:30.496152 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:30.496116 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" podUID="801c5378-127c-49f6-8bea-562ca9ccb4c1" containerName="limitador" containerID="cri-o://7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e" gracePeriod=30 Jul 22 11:33:31.038737 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.038716 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:31.107119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.107064 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file\") pod \"801c5378-127c-49f6-8bea-562ca9ccb4c1\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " Jul 22 11:33:31.107210 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.107153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5cwv\" (UniqueName: \"kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv\") pod \"801c5378-127c-49f6-8bea-562ca9ccb4c1\" (UID: \"801c5378-127c-49f6-8bea-562ca9ccb4c1\") " Jul 22 11:33:31.107373 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.107353 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file" (OuterVolumeSpecName: "config-file") pod "801c5378-127c-49f6-8bea-562ca9ccb4c1" (UID: "801c5378-127c-49f6-8bea-562ca9ccb4c1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:33:31.109208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.109187 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv" (OuterVolumeSpecName: "kube-api-access-c5cwv") pod "801c5378-127c-49f6-8bea-562ca9ccb4c1" (UID: "801c5378-127c-49f6-8bea-562ca9ccb4c1"). InnerVolumeSpecName "kube-api-access-c5cwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:33:31.208041 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.208006 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cwv\" (UniqueName: \"kubernetes.io/projected/801c5378-127c-49f6-8bea-562ca9ccb4c1-kube-api-access-c5cwv\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:33:31.208041 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.208038 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/801c5378-127c-49f6-8bea-562ca9ccb4c1-config-file\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:33:31.810685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.810656 2569 generic.go:356] "Generic (PLEG): container finished" podID="801c5378-127c-49f6-8bea-562ca9ccb4c1" containerID="7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e" exitCode=0 Jul 22 11:33:31.810807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.810702 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" Jul 22 11:33:31.810807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.810713 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" event={"ID":"801c5378-127c-49f6-8bea-562ca9ccb4c1","Type":"ContainerDied","Data":"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e"} Jul 22 11:33:31.810807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.810759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x7tv2" event={"ID":"801c5378-127c-49f6-8bea-562ca9ccb4c1","Type":"ContainerDied","Data":"2a2462c7735792e57fcfe8da489382497767613fe6b46d02fc9a90eed248dad3"} Jul 22 11:33:31.810807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.810787 2569 scope.go:117] "RemoveContainer" containerID="7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e" Jul 22 11:33:31.825112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.825089 2569 scope.go:117] "RemoveContainer" containerID="7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e" Jul 22 11:33:31.825338 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:33:31.825320 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e\": container with ID starting with 7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e not found: ID does not exist" containerID="7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e" Jul 22 11:33:31.825388 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.825346 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e"} err="failed to get container status \"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e\": rpc error: code = NotFound desc = could not find container \"7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e\": container with ID starting with 7d9548a484bafe21509d95efe1209f2a8d231dbce2539dd42e603c226986104e not found: ID does not exist" Jul 22 11:33:31.842846 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.842822 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:31.849079 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:31.849060 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x7tv2"] Jul 22 11:33:32.349016 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:32.348994 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801c5378-127c-49f6-8bea-562ca9ccb4c1" path="/var/lib/kubelet/pods/801c5378-127c-49f6-8bea-562ca9ccb4c1/volumes" Jul 22 11:33:34.595277 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:34.595245 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:34.756524 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:34.756499 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:35.053397 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.053370 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ncktw"] Jul 22 11:33:35.053652 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.053640 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801c5378-127c-49f6-8bea-562ca9ccb4c1" containerName="limitador" Jul 22 11:33:35.053693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.053654 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="801c5378-127c-49f6-8bea-562ca9ccb4c1" containerName="limitador" Jul 22 11:33:35.053727 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.053715 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="801c5378-127c-49f6-8bea-562ca9ccb4c1" containerName="limitador" Jul 22 11:33:35.056536 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.056522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.060298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.060283 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-76cxh\"" Jul 22 11:33:35.060677 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.060663 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 11:33:35.060984 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.060948 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 11:33:35.061064 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.060989 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 11:33:35.083559 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.083534 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ncktw"] Jul 22 11:33:35.140237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.140219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnjk\" (UniqueName: \"kubernetes.io/projected/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-kube-api-access-jdnjk\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.140332 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.140265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-data\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.141322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.141304 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:35.241232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.241208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-data\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.241324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.241261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnjk\" (UniqueName: \"kubernetes.io/projected/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-kube-api-access-jdnjk\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.241567 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.241552 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-data\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.251947 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.251915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnjk\" (UniqueName: \"kubernetes.io/projected/79383aa6-ad3b-4a6d-bfc6-665e2b15a653-kube-api-access-jdnjk\") pod \"postgres-69fb4594cf-ncktw\" (UID: \"79383aa6-ad3b-4a6d-bfc6-665e2b15a653\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.366629 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.366574 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:35.497293 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.497267 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ncktw"] Jul 22 11:33:35.500593 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:33:35.500567 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79383aa6_ad3b_4a6d_bfc6_665e2b15a653.slice/crio-baaa4bf1bdafcff224457ec65f0a4a04d592442302e3facd3ee131a57373e0ce WatchSource:0}: Error finding container baaa4bf1bdafcff224457ec65f0a4a04d592442302e3facd3ee131a57373e0ce: Status 404 returned error can't find the container with id baaa4bf1bdafcff224457ec65f0a4a04d592442302e3facd3ee131a57373e0ce Jul 22 11:33:35.828555 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:35.828523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" event={"ID":"79383aa6-ad3b-4a6d-bfc6-665e2b15a653","Type":"ContainerStarted","Data":"baaa4bf1bdafcff224457ec65f0a4a04d592442302e3facd3ee131a57373e0ce"} Jul 22 11:33:40.855748 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:40.855723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" event={"ID":"79383aa6-ad3b-4a6d-bfc6-665e2b15a653","Type":"ContainerStarted","Data":"3adc94a9b54b42e0fa14303011f1525c752f3df5ecd0403945f4e23b361f137d"} Jul 22 11:33:40.856105 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:40.855824 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:40.880479 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:40.880439 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" podStartSLOduration=0.60770382 podStartE2EDuration="5.880426843s" totalImagesPullingTime="5.272723023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:33:40.878878211 +0000 UTC m=+591.040420933" watchObservedRunningTime="2026-07-22 11:33:40.880426843 +0000 UTC m=+591.041969537" Jul 22 11:33:46.888342 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:46.888316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ncktw" Jul 22 11:33:47.353843 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:47.353815 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:47.611841 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:47.611752 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:50.226549 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:50.226518 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:33:50.228715 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:50.228695 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:33:52.457033 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:52.457000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:55.415064 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.415020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:55.427943 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.427916 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:33:55.445209 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.445186 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.448367 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.448344 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52s5k\"" Jul 22 11:33:55.448521 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.448403 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 11:33:55.448924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.448904 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 11:33:55.450940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.450919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:33:55.540164 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.540135 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:55.595900 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.595875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2rl\" (UniqueName: \"kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.596033 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.595947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.631395 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.631374 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:55.631481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.631401 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:33:55.633902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.633888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.648632 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.648612 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:33:55.696770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.696723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfs49\" (UniqueName: \"kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.696770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.696755 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2rl\" (UniqueName: \"kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.696920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.696832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.696920 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.696865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.697044 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:33:55.696946 2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 22 11:33:55.697044 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:33:55.697033 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert podName:c54ddf1e-0956-468a-8ae7-14d96e072814 nodeName:}" failed. No retries permitted until 2026-07-22 11:33:56.197015347 +0000 UTC m=+606.358558021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert") pod "maas-controller-5bc584f68f-9rpmq" (UID: "c54ddf1e-0956-468a-8ae7-14d96e072814") : secret "maas-controller-webhook-cert" not found Jul 22 11:33:55.706678 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.706656 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2rl\" (UniqueName: \"kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:55.797445 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.797422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.797531 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.797459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfs49\" (UniqueName: \"kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.799896 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.799876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.806469 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.806449 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfs49\" (UniqueName: \"kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49\") pod \"maas-controller-74bd8b57c-dfc9h\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:55.835993 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.835969 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:55.943240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:55.943218 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:56.201135 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.201105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:56.203458 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.203415 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") pod \"maas-controller-5bc584f68f-9rpmq\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:56.274469 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.274449 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:33:56.276591 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:33:56.276566 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c4ef7_a017_4761_9bcd_a73ea4944d62.slice/crio-40f77a19fc45514608b9210cfa56f732330dc955d68c1e9ec451787b8c51610d WatchSource:0}: Error finding container 40f77a19fc45514608b9210cfa56f732330dc955d68c1e9ec451787b8c51610d: Status 404 returned error can't find the container with id 40f77a19fc45514608b9210cfa56f732330dc955d68c1e9ec451787b8c51610d Jul 22 11:33:56.356824 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.356806 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:56.502888 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.502706 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:33:56.505437 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:33:56.505409 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ddf1e_0956_468a_8ae7_14d96e072814.slice/crio-88235d971d8c40510b19409a87302430ac1137e3c5a597d962e0075b82bd3eba WatchSource:0}: Error finding container 88235d971d8c40510b19409a87302430ac1137e3c5a597d962e0075b82bd3eba: Status 404 returned error can't find the container with id 88235d971d8c40510b19409a87302430ac1137e3c5a597d962e0075b82bd3eba Jul 22 11:33:56.511506 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.511480 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:56.595385 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.595357 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:56.925546 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.925509 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" event={"ID":"654c4ef7-a017-4761-9bcd-a73ea4944d62","Type":"ContainerStarted","Data":"40f77a19fc45514608b9210cfa56f732330dc955d68c1e9ec451787b8c51610d"} Jul 22 11:33:56.926932 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:56.926896 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" event={"ID":"c54ddf1e-0956-468a-8ae7-14d96e072814","Type":"ContainerStarted","Data":"88235d971d8c40510b19409a87302430ac1137e3c5a597d962e0075b82bd3eba"} Jul 22 11:33:58.938485 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.938451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" event={"ID":"c54ddf1e-0956-468a-8ae7-14d96e072814","Type":"ContainerStarted","Data":"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526"} Jul 22 11:33:58.938867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.938514 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:33:58.939901 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.939881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" event={"ID":"654c4ef7-a017-4761-9bcd-a73ea4944d62","Type":"ContainerStarted","Data":"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d"} Jul 22 11:33:58.940002 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.939990 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:33:58.961030 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.960985 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" podStartSLOduration=1.635882375 podStartE2EDuration="3.960948542s" totalImagesPullingTime="2.325066167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:33:58.958619347 +0000 UTC m=+609.120162085" watchObservedRunningTime="2026-07-22 11:33:58.960948542 +0000 UTC m=+609.122491423" Jul 22 11:33:58.989026 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:58.988973 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" podStartSLOduration=1.441541067 podStartE2EDuration="3.988941232s" totalImagesPullingTime="2.547400165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:33:58.986348651 +0000 UTC m=+609.147891340" watchObservedRunningTime="2026-07-22 11:33:58.988941232 +0000 UTC m=+609.150483943" Jul 22 11:33:59.015180 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:59.015154 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:33:59.118414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:33:59.118336 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:09.947385 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:09.947356 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:34:09.949922 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:09.947428 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:34:10.018506 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.018467 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:34:10.018714 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.018669 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" podUID="c54ddf1e-0956-468a-8ae7-14d96e072814" containerName="manager" containerID="cri-o://cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526" gracePeriod=30 Jul 22 11:34:10.280589 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.280566 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:34:10.406714 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.406658 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") pod \"c54ddf1e-0956-468a-8ae7-14d96e072814\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " Jul 22 11:34:10.406838 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.406715 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf2rl\" (UniqueName: \"kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl\") pod \"c54ddf1e-0956-468a-8ae7-14d96e072814\" (UID: \"c54ddf1e-0956-468a-8ae7-14d96e072814\") " Jul 22 11:34:10.408833 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.408808 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c54ddf1e-0956-468a-8ae7-14d96e072814" (UID: "c54ddf1e-0956-468a-8ae7-14d96e072814"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:34:10.408980 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.408833 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl" (OuterVolumeSpecName: "kube-api-access-bf2rl") pod "c54ddf1e-0956-468a-8ae7-14d96e072814" (UID: "c54ddf1e-0956-468a-8ae7-14d96e072814"). InnerVolumeSpecName "kube-api-access-bf2rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:34:10.507478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.507447 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c54ddf1e-0956-468a-8ae7-14d96e072814-webhook-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:34:10.507478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.507471 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2rl\" (UniqueName: \"kubernetes.io/projected/c54ddf1e-0956-468a-8ae7-14d96e072814-kube-api-access-bf2rl\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:34:10.993498 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.993471 2569 generic.go:356] "Generic (PLEG): container finished" podID="c54ddf1e-0956-468a-8ae7-14d96e072814" containerID="cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526" exitCode=0 Jul 22 11:34:10.993857 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.993528 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" Jul 22 11:34:10.993857 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.993563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" event={"ID":"c54ddf1e-0956-468a-8ae7-14d96e072814","Type":"ContainerDied","Data":"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526"} Jul 22 11:34:10.993857 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.993615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc584f68f-9rpmq" event={"ID":"c54ddf1e-0956-468a-8ae7-14d96e072814","Type":"ContainerDied","Data":"88235d971d8c40510b19409a87302430ac1137e3c5a597d962e0075b82bd3eba"} Jul 22 11:34:10.993857 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:10.993635 2569 scope.go:117] "RemoveContainer" containerID="cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526" Jul 22 11:34:11.002495 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:11.002481 2569 scope.go:117] "RemoveContainer" containerID="cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526" Jul 22 11:34:11.002753 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:34:11.002734 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526\": container with ID starting with cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526 not found: ID does not exist" containerID="cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526" Jul 22 11:34:11.002791 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:11.002762 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526"} err="failed to get container status \"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526\": rpc error: code = NotFound desc = could not find container \"cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526\": container with ID starting with cbc3444c19910402654d249e4c4f17b13b882013560628dc887906dc09140526 not found: ID does not exist" Jul 22 11:34:11.018021 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:11.017990 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:34:11.022008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:11.021988 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bc584f68f-9rpmq"] Jul 22 11:34:12.349072 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:12.349036 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54ddf1e-0956-468a-8ae7-14d96e072814" path="/var/lib/kubelet/pods/c54ddf1e-0956-468a-8ae7-14d96e072814/volumes" Jul 22 11:34:31.826019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:31.825982 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:32.611006 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.610972 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d896bf67-vws88"] Jul 22 11:34:32.611761 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.611740 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54ddf1e-0956-468a-8ae7-14d96e072814" containerName="manager" Jul 22 11:34:32.611872 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.611764 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54ddf1e-0956-468a-8ae7-14d96e072814" containerName="manager" Jul 22 11:34:32.611925 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.611912 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54ddf1e-0956-468a-8ae7-14d96e072814" containerName="manager" Jul 22 11:34:32.620535 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.620484 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.625489 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.625166 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rrgrs\"" Jul 22 11:34:32.626189 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.626167 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 11:34:32.629045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.629021 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d896bf67-vws88"] Jul 22 11:34:32.778315 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.778275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.778511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.778362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvj6w\" (UniqueName: \"kubernetes.io/projected/c80c1a97-201f-43df-b995-769796fc8bed-kube-api-access-lvj6w\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.778511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.778410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80c1a97-201f-43df-b995-769796fc8bed-openshift-service-ca\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.879520 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.879447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvj6w\" (UniqueName: \"kubernetes.io/projected/c80c1a97-201f-43df-b995-769796fc8bed-kube-api-access-lvj6w\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.879520 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.879504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80c1a97-201f-43df-b995-769796fc8bed-openshift-service-ca\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.880048 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.879548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.880048 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:34:32.879676 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 22 11:34:32.880048 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:34:32.879761 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls podName:c80c1a97-201f-43df-b995-769796fc8bed nodeName:}" failed. No retries permitted until 2026-07-22 11:34:33.379737221 +0000 UTC m=+643.541279900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls") pod "maas-api-58d896bf67-vws88" (UID: "c80c1a97-201f-43df-b995-769796fc8bed") : secret "maas-api-serving-cert" not found Jul 22 11:34:32.880302 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.880273 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80c1a97-201f-43df-b995-769796fc8bed-openshift-service-ca\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:32.890909 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:32.890886 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvj6w\" (UniqueName: \"kubernetes.io/projected/c80c1a97-201f-43df-b995-769796fc8bed-kube-api-access-lvj6w\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:33.304043 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.304013 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zprxj"] Jul 22 11:34:33.307719 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.307704 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.309940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.309921 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 11:34:33.320400 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.320381 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zprxj"] Jul 22 11:34:33.383300 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.383268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtx2\" (UniqueName: \"kubernetes.io/projected/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-kube-api-access-lrtx2\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.383428 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.383342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.383428 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.383399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:33.385816 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.385793 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80c1a97-201f-43df-b995-769796fc8bed-maas-api-tls\") pod \"maas-api-58d896bf67-vws88\" (UID: \"c80c1a97-201f-43df-b995-769796fc8bed\") " pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:33.484596 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.484566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtx2\" (UniqueName: \"kubernetes.io/projected/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-kube-api-access-lrtx2\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.484770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.484659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.485349 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.485331 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.494376 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.494352 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtx2\" (UniqueName: \"kubernetes.io/projected/761f94f3-af3c-4bea-9122-40ba7a8a2ec0-kube-api-access-lrtx2\") pod \"payload-pre-processing-7677b774bb-zprxj\" (UID: \"761f94f3-af3c-4bea-9122-40ba7a8a2ec0\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.532256 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.532226 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:33.617775 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.617746 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:33.674559 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.674524 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d896bf67-vws88"] Jul 22 11:34:33.757129 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:33.757091 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zprxj"] Jul 22 11:34:33.759609 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:34:33.759582 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761f94f3_af3c_4bea_9122_40ba7a8a2ec0.slice/crio-88f4a81f4f4c1cbf492ffe6887320fac4dde041b8e0fb950e1737c7a385b4ee8 WatchSource:0}: Error finding container 88f4a81f4f4c1cbf492ffe6887320fac4dde041b8e0fb950e1737c7a385b4ee8: Status 404 returned error can't find the container with id 88f4a81f4f4c1cbf492ffe6887320fac4dde041b8e0fb950e1737c7a385b4ee8 Jul 22 11:34:34.046699 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:34.046615 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:34.095637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:34.095589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" event={"ID":"c80c1a97-201f-43df-b995-769796fc8bed","Type":"ContainerStarted","Data":"b3b54fb4238c31d309defae55995e52f339cce2a16ec4b9b5d5b922468c8ce00"} Jul 22 11:34:34.097344 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:34.097310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" event={"ID":"761f94f3-af3c-4bea-9122-40ba7a8a2ec0","Type":"ContainerStarted","Data":"88f4a81f4f4c1cbf492ffe6887320fac4dde041b8e0fb950e1737c7a385b4ee8"} Jul 22 11:34:36.112650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:36.112623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" event={"ID":"c80c1a97-201f-43df-b995-769796fc8bed","Type":"ContainerStarted","Data":"78138b9b4e37a4079fe5f788bc09443fad5128b0e0bc3c12187ea76634a799f9"} Jul 22 11:34:36.112971 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:36.112684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:36.134800 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:36.134762 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" podStartSLOduration=2.281065097 podStartE2EDuration="4.134747393s" totalImagesPullingTime="1.853682296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:34:36.132685605 +0000 UTC m=+646.294228300" watchObservedRunningTime="2026-07-22 11:34:36.134747393 +0000 UTC m=+646.296290088" Jul 22 11:34:37.118574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:37.118536 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" event={"ID":"761f94f3-af3c-4bea-9122-40ba7a8a2ec0","Type":"ContainerStarted","Data":"f04b1de83e758d7dcc4178fd9c51a022e5570317594343b9141cbd394b6ea3d9"} Jul 22 11:34:37.118974 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:37.118808 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:41.136878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.136851 2569 generic.go:356] "Generic (PLEG): container finished" podID="d412bde1-a4b9-4024-9815-879a50c84655" containerID="162b421d181f8317a4720703708e858989af38d26c9c2df63bbaff8ddabb0736" exitCode=0 Jul 22 11:34:41.137227 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.136920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" event={"ID":"d412bde1-a4b9-4024-9815-879a50c84655","Type":"ContainerDied","Data":"162b421d181f8317a4720703708e858989af38d26c9c2df63bbaff8ddabb0736"} Jul 22 11:34:41.137292 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.137275 2569 scope.go:117] "RemoveContainer" containerID="162b421d181f8317a4720703708e858989af38d26c9c2df63bbaff8ddabb0736" Jul 22 11:34:41.138243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.138168 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:34:41.138243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.138230 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:34:41.138383 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.138262 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:34:41.159471 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:41.159433 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" podStartSLOduration=5.852268214 podStartE2EDuration="8.159418347s" totalImagesPullingTime="2.307150133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:34:37.136626165 +0000 UTC m=+647.298168878" watchObservedRunningTime="2026-07-22 11:34:41.159418347 +0000 UTC m=+651.320961061" Jul 22 11:34:42.124626 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:42.124600 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d896bf67-vws88" Jul 22 11:34:42.142724 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:42.142692 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" event={"ID":"d412bde1-a4b9-4024-9815-879a50c84655","Type":"ContainerStarted","Data":"edb7cb0819adfdddfd729c69fa2f55f021978a54b29f2188c26013d372185212"} Jul 22 11:34:42.494671 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:42.494576 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:34:42.495060 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:42.494971 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-zl9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 22 11:34:42.495245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:42.495222 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" podUID="d412bde1-a4b9-4024-9815-879a50c84655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 22 11:34:43.190515 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:43.190475 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:43.299905 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:43.299868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:43.494488 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:43.494405 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-zl9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 22 11:34:43.494735 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:43.494707 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" podUID="d412bde1-a4b9-4024-9815-879a50c84655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 22 11:34:44.494525 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:44.494486 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-zl9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 22 11:34:44.495112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:44.495081 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" podUID="d412bde1-a4b9-4024-9815-879a50c84655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 22 11:34:45.494832 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:45.494797 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-zl9g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 22 11:34:45.495445 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:45.495414 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" podUID="d412bde1-a4b9-4024-9815-879a50c84655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 22 11:34:46.497761 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:46.497733 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:34:46.498132 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:46.497989 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:34:46.498586 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:46.498566 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-zl9g6" Jul 22 11:34:48.125527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:48.125495 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-zprxj" Jul 22 11:34:52.396188 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.396151 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw"] Jul 22 11:34:52.400235 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.400217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.403088 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.403058 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 11:34:52.403220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.403104 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 11:34:52.403220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.403113 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 11:34:52.403220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.403172 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t5vvq\"" Jul 22 11:34:52.409867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.409846 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw"] Jul 22 11:34:52.417156 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.417131 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:34:52.447144 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91075186-e772-4761-9683-8e30ed4dcbb2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.447249 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.447310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7gm\" (UniqueName: \"kubernetes.io/projected/91075186-e772-4761-9683-8e30ed4dcbb2-kube-api-access-bc7gm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.447310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447301 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.447419 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.447419 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.447396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548343 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7gm\" (UniqueName: \"kubernetes.io/projected/91075186-e772-4761-9683-8e30ed4dcbb2-kube-api-access-bc7gm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91075186-e772-4761-9683-8e30ed4dcbb2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548717 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548868 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548936 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.548936 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.548927 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.550750 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.550730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91075186-e772-4761-9683-8e30ed4dcbb2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.551262 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.551241 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91075186-e772-4761-9683-8e30ed4dcbb2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.559390 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.559339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7gm\" (UniqueName: \"kubernetes.io/projected/91075186-e772-4761-9683-8e30ed4dcbb2-kube-api-access-bc7gm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw\" (UID: \"91075186-e772-4761-9683-8e30ed4dcbb2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.710024 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.709921 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:34:52.848181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.848149 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw"] Jul 22 11:34:52.851405 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:34:52.851378 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91075186_e772_4761_9683_8e30ed4dcbb2.slice/crio-c463e8c07a580f7e5bf8697dadb28ab8b9fefd650480210f13ac03633a8247e9 WatchSource:0}: Error finding container c463e8c07a580f7e5bf8697dadb28ab8b9fefd650480210f13ac03633a8247e9: Status 404 returned error can't find the container with id c463e8c07a580f7e5bf8697dadb28ab8b9fefd650480210f13ac03633a8247e9 Jul 22 11:34:52.859172 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:52.859146 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:34:53.196303 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:53.196264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" event={"ID":"91075186-e772-4761-9683-8e30ed4dcbb2","Type":"ContainerStarted","Data":"c463e8c07a580f7e5bf8697dadb28ab8b9fefd650480210f13ac03633a8247e9"} Jul 22 11:34:54.202503 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:54.202468 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" event={"ID":"91075186-e772-4761-9683-8e30ed4dcbb2","Type":"ContainerStarted","Data":"08cf661e302599f60bc97be74738a81395d0f9b89b4dcbd870e54735f7824f12"} Jul 22 11:34:54.223087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:54.223040 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" podStartSLOduration=1.04605088 podStartE2EDuration="2.223024826s" totalImagesPullingTime="1.176973946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:34:54.220608452 +0000 UTC m=+664.382151146" watchObservedRunningTime="2026-07-22 11:34:54.223024826 +0000 UTC m=+664.384567520" Jul 22 11:34:59.942221 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:34:59.942181 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:35:02.710267 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:02.710225 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:35:02.710733 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:02.710300 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:35:02.722684 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:02.722649 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:35:03.253212 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:03.253184 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw" Jul 22 11:35:15.183277 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.183248 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59"] Jul 22 11:35:15.186369 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.186334 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.189025 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.189004 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 11:35:15.198799 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.198779 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59"] Jul 22 11:35:15.227632 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.227605 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:35:15.348729 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348699 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jff2k\" (UniqueName: \"kubernetes.io/projected/a3988074-8ec4-49d6-a83e-e235d941e3cb-kube-api-access-jff2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.348729 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.348983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.348983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.348983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.348983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.348967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3988074-8ec4-49d6-a83e-e235d941e3cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450044 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.449985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450044 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450211 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450211 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450337 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3988074-8ec4-49d6-a83e-e235d941e3cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450402 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jff2k\" (UniqueName: \"kubernetes.io/projected/a3988074-8ec4-49d6-a83e-e235d941e3cb-kube-api-access-jff2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450446 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450593 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.450865 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.450842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.452417 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.452400 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3988074-8ec4-49d6-a83e-e235d941e3cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.452657 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.452641 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3988074-8ec4-49d6-a83e-e235d941e3cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.460715 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.460691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jff2k\" (UniqueName: \"kubernetes.io/projected/a3988074-8ec4-49d6-a83e-e235d941e3cb-kube-api-access-jff2k\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p5r59\" (UID: \"a3988074-8ec4-49d6-a83e-e235d941e3cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.496836 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.496815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:15.624501 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:15.624477 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59"] Jul 22 11:35:15.626203 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:35:15.626162 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3988074_8ec4_49d6_a83e_e235d941e3cb.slice/crio-e48e9c8edf3a1c0e5666193f130ee22baab9d8f858fb056ba65b6996b557d4d8 WatchSource:0}: Error finding container e48e9c8edf3a1c0e5666193f130ee22baab9d8f858fb056ba65b6996b557d4d8: Status 404 returned error can't find the container with id e48e9c8edf3a1c0e5666193f130ee22baab9d8f858fb056ba65b6996b557d4d8 Jul 22 11:35:16.300610 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:16.300540 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" event={"ID":"a3988074-8ec4-49d6-a83e-e235d941e3cb","Type":"ContainerStarted","Data":"02424af4f1e5d57aaaad2ce6e07d1ab650d6c67f3a802dc2f698b1686ef096ac"} Jul 22 11:35:16.301031 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:16.300687 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" event={"ID":"a3988074-8ec4-49d6-a83e-e235d941e3cb","Type":"ContainerStarted","Data":"e48e9c8edf3a1c0e5666193f130ee22baab9d8f858fb056ba65b6996b557d4d8"} Jul 22 11:35:16.320795 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:16.320750 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" podStartSLOduration=0.977063617 podStartE2EDuration="1.320735594s" totalImagesPullingTime="343.671977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:35:16.318416689 +0000 UTC m=+686.479959383" watchObservedRunningTime="2026-07-22 11:35:16.320735594 +0000 UTC m=+686.482278288" Jul 22 11:35:25.497923 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:25.497891 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:25.498349 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:25.497939 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:25.510305 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:25.510280 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:26.356620 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:26.356597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p5r59" Jul 22 11:35:26.454246 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:26.454219 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:35:34.442275 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.442245 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g"] Jul 22 11:35:34.445653 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.445634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.447839 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:35:34.447798 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" podUID="6e05f0ea-63ab-4de4-8f50-1a7d51b10989" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.448923 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:35:34.448887 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"facebook-opt-125m-simulated-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-136-40.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-136-40.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" Jul 22 11:35:34.462405 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.462381 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g"] Jul 22 11:35:34.471456 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.471428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:35:34.506259 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.506364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.506364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t42r\" (UniqueName: \"kubernetes.io/projected/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-kube-api-access-5t42r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.506364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.506568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.506568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.506477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607252 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607252 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t42r\" (UniqueName: \"kubernetes.io/projected/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-kube-api-access-5t42r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607456 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607456 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607456 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607456 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607691 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607671 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607745 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.607745 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.607713 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.609651 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.609629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:34.618257 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:34.618235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t42r\" (UniqueName: \"kubernetes.io/projected/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-kube-api-access-5t42r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:35.608365 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:35:35.608332 2569 secret.go:189] Couldn't get secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 22 11:35:35.608680 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:35:35.608399 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs podName:6e05f0ea-63ab-4de4-8f50-1a7d51b10989 nodeName:}" failed. No retries permitted until 2026-07-22 11:35:36.108384194 +0000 UTC m=+706.269926868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs") pod "facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" (UID: "6e05f0ea-63ab-4de4-8f50-1a7d51b10989") : failed to sync secret cache: timed out waiting for the condition Jul 22 11:35:35.821511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:35.821483 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 11:35:36.117907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:36.117877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:36.120391 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:36.120362 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e05f0ea-63ab-4de4-8f50-1a7d51b10989-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4bp6g\" (UID: \"6e05f0ea-63ab-4de4-8f50-1a7d51b10989\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:36.260913 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:36.260890 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:36.396381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:36.396355 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g"] Jul 22 11:35:36.398539 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:35:36.398512 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e05f0ea_63ab_4de4_8f50_1a7d51b10989.slice/crio-75dc05f5766fd7c3e1ad0448585b259fd56ffcb8c82fa69ea48dc440fd8c8a20 WatchSource:0}: Error finding container 75dc05f5766fd7c3e1ad0448585b259fd56ffcb8c82fa69ea48dc440fd8c8a20: Status 404 returned error can't find the container with id 75dc05f5766fd7c3e1ad0448585b259fd56ffcb8c82fa69ea48dc440fd8c8a20 Jul 22 11:35:37.395914 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:37.395883 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" event={"ID":"6e05f0ea-63ab-4de4-8f50-1a7d51b10989","Type":"ContainerStarted","Data":"3f586ed652c25fa199ce1dd0e4d10cfa86e2e3409fb292eebeac78563a1f4b3c"} Jul 22 11:35:37.396272 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:37.395926 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" event={"ID":"6e05f0ea-63ab-4de4-8f50-1a7d51b10989","Type":"ContainerStarted","Data":"75dc05f5766fd7c3e1ad0448585b259fd56ffcb8c82fa69ea48dc440fd8c8a20"} Jul 22 11:35:37.414772 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:37.414722 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" podStartSLOduration=3.156439572 podStartE2EDuration="3.414702543s" totalImagesPullingTime="258.262971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:35:37.414008009 +0000 UTC m=+707.575550721" watchObservedRunningTime="2026-07-22 11:35:37.414702543 +0000 UTC m=+707.576245240" Jul 22 11:35:46.261718 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:46.261684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:46.262095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:46.261753 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:46.273230 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:46.273210 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:46.444842 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:46.444813 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4bp6g" Jul 22 11:35:49.243050 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:35:49.243020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:24.124568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:36:24.124536 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:40.507424 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:36:40.507386 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:44.076788 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:36:44.076754 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:14.985186 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:14.985090 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:15.086878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:15.086835 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:16.208889 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:16.208852 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:16.302005 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:16.301974 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:37.655802 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:37.655761 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:37.726186 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:37:37.726151 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:38:04.705929 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:04.705895 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:38:04.706474 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:04.706092 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" podUID="654c4ef7-a017-4761-9bcd-a73ea4944d62" containerName="manager" containerID="cri-o://b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d" gracePeriod=30 Jul 22 11:38:04.949482 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:04.949459 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:38:05.012787 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.012720 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert\") pod \"654c4ef7-a017-4761-9bcd-a73ea4944d62\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " Jul 22 11:38:05.012916 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.012832 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfs49\" (UniqueName: \"kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49\") pod \"654c4ef7-a017-4761-9bcd-a73ea4944d62\" (UID: \"654c4ef7-a017-4761-9bcd-a73ea4944d62\") " Jul 22 11:38:05.014927 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.014901 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "654c4ef7-a017-4761-9bcd-a73ea4944d62" (UID: "654c4ef7-a017-4761-9bcd-a73ea4944d62"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:38:05.015065 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.014949 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49" (OuterVolumeSpecName: "kube-api-access-qfs49") pod "654c4ef7-a017-4761-9bcd-a73ea4944d62" (UID: "654c4ef7-a017-4761-9bcd-a73ea4944d62"). InnerVolumeSpecName "kube-api-access-qfs49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:38:05.037162 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.037135 2569 generic.go:356] "Generic (PLEG): container finished" podID="654c4ef7-a017-4761-9bcd-a73ea4944d62" containerID="b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d" exitCode=0 Jul 22 11:38:05.037261 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.037186 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" Jul 22 11:38:05.037261 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.037222 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" event={"ID":"654c4ef7-a017-4761-9bcd-a73ea4944d62","Type":"ContainerDied","Data":"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d"} Jul 22 11:38:05.037376 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.037282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-dfc9h" event={"ID":"654c4ef7-a017-4761-9bcd-a73ea4944d62","Type":"ContainerDied","Data":"40f77a19fc45514608b9210cfa56f732330dc955d68c1e9ec451787b8c51610d"} Jul 22 11:38:05.037376 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.037317 2569 scope.go:117] "RemoveContainer" containerID="b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d" Jul 22 11:38:05.046046 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.046029 2569 scope.go:117] "RemoveContainer" containerID="b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d" Jul 22 11:38:05.046299 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:38:05.046281 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d\": container with ID starting with b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d not found: ID does not exist" containerID="b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d" Jul 22 11:38:05.046373 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.046307 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d"} err="failed to get container status \"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d\": rpc error: code = NotFound desc = could not find container \"b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d\": container with ID starting with b1e2f7f8777521d76f8040bdee6418bec96e00eeebd1e7c916e58061e1e2a43d not found: ID does not exist" Jul 22 11:38:05.058866 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.058844 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:38:05.063237 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.063218 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-dfc9h"] Jul 22 11:38:05.114344 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.114323 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfs49\" (UniqueName: \"kubernetes.io/projected/654c4ef7-a017-4761-9bcd-a73ea4944d62-kube-api-access-qfs49\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:38:05.114344 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:05.114345 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/654c4ef7-a017-4761-9bcd-a73ea4944d62-webhook-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:38:06.354919 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:06.354885 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654c4ef7-a017-4761-9bcd-a73ea4944d62" path="/var/lib/kubelet/pods/654c4ef7-a017-4761-9bcd-a73ea4944d62/volumes" Jul 22 11:38:50.255134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:50.255106 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:38:50.255685 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:38:50.255407 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:39:05.945886 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.945850 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:05.946357 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.946245 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654c4ef7-a017-4761-9bcd-a73ea4944d62" containerName="manager" Jul 22 11:39:05.946357 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.946260 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="654c4ef7-a017-4761-9bcd-a73ea4944d62" containerName="manager" Jul 22 11:39:05.946357 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.946326 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="654c4ef7-a017-4761-9bcd-a73ea4944d62" containerName="manager" Jul 22 11:39:05.949248 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.949232 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:05.952008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.951984 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 11:39:05.952186 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.952090 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 11:39:05.952186 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.952138 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52s5k\"" Jul 22 11:39:05.960267 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:05.960246 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:06.048480 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.048457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znh7x\" (UniqueName: \"kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.048598 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.048517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.149249 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.149222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.149363 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.149278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znh7x\" (UniqueName: \"kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.151693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.151675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.160809 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.160789 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znh7x\" (UniqueName: \"kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x\") pod \"maas-controller-74bd8b57c-pxq9s\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.259661 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.259608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:06.396229 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:06.396208 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:06.398237 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:39:06.398208 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8def7913_f8e7_44e8_a65f_be8a30fff816.slice/crio-dd27f2d0c748078b357bb612a040a16433fb5c50ca93294246335e44d6d591e0 WatchSource:0}: Error finding container dd27f2d0c748078b357bb612a040a16433fb5c50ca93294246335e44d6d591e0: Status 404 returned error can't find the container with id dd27f2d0c748078b357bb612a040a16433fb5c50ca93294246335e44d6d591e0 Jul 22 11:39:07.337055 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:07.336999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" event={"ID":"8def7913-f8e7-44e8-a65f-be8a30fff816","Type":"ContainerStarted","Data":"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216"} Jul 22 11:39:07.337451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:07.337112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" event={"ID":"8def7913-f8e7-44e8-a65f-be8a30fff816","Type":"ContainerStarted","Data":"dd27f2d0c748078b357bb612a040a16433fb5c50ca93294246335e44d6d591e0"} Jul 22 11:39:07.337451 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:07.337145 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:07.365763 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:07.365698 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" podStartSLOduration=1.746801288 podStartE2EDuration="2.365684211s" totalImagesPullingTime="618.882923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:39:07.364748092 +0000 UTC m=+917.526290804" watchObservedRunningTime="2026-07-22 11:39:07.365684211 +0000 UTC m=+917.527226909" Jul 22 11:39:18.349511 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:18.349481 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:23.066998 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.066941 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:39:23.083331 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.083305 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:23.083555 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.083533 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" podUID="8def7913-f8e7-44e8-a65f-be8a30fff816" containerName="manager" containerID="cri-o://e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216" gracePeriod=30 Jul 22 11:39:23.330398 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.330342 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:23.397399 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.397372 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znh7x\" (UniqueName: \"kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x\") pod \"8def7913-f8e7-44e8-a65f-be8a30fff816\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " Jul 22 11:39:23.397585 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.397417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert\") pod \"8def7913-f8e7-44e8-a65f-be8a30fff816\" (UID: \"8def7913-f8e7-44e8-a65f-be8a30fff816\") " Jul 22 11:39:23.399613 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.399589 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8def7913-f8e7-44e8-a65f-be8a30fff816" (UID: "8def7913-f8e7-44e8-a65f-be8a30fff816"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:39:23.399733 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.399691 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x" (OuterVolumeSpecName: "kube-api-access-znh7x") pod "8def7913-f8e7-44e8-a65f-be8a30fff816" (UID: "8def7913-f8e7-44e8-a65f-be8a30fff816"). InnerVolumeSpecName "kube-api-access-znh7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:39:23.412448 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.412419 2569 generic.go:356] "Generic (PLEG): container finished" podID="8def7913-f8e7-44e8-a65f-be8a30fff816" containerID="e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216" exitCode=0 Jul 22 11:39:23.412540 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.412461 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" Jul 22 11:39:23.412540 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.412480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" event={"ID":"8def7913-f8e7-44e8-a65f-be8a30fff816","Type":"ContainerDied","Data":"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216"} Jul 22 11:39:23.412540 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.412523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-pxq9s" event={"ID":"8def7913-f8e7-44e8-a65f-be8a30fff816","Type":"ContainerDied","Data":"dd27f2d0c748078b357bb612a040a16433fb5c50ca93294246335e44d6d591e0"} Jul 22 11:39:23.412658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.412544 2569 scope.go:117] "RemoveContainer" containerID="e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216" Jul 22 11:39:23.421058 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.421041 2569 scope.go:117] "RemoveContainer" containerID="e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216" Jul 22 11:39:23.421336 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:39:23.421316 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216\": container with ID starting with e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216 not found: ID does not exist" containerID="e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216" Jul 22 11:39:23.421385 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.421345 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216"} err="failed to get container status \"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216\": rpc error: code = NotFound desc = could not find container \"e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216\": container with ID starting with e9d4b0f839ca9c6edf235e167ee19e624560c14d9c59fef1554e5d040c7c2216 not found: ID does not exist" Jul 22 11:39:23.437182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.437159 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:23.452976 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.451051 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-pxq9s"] Jul 22 11:39:23.498924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.498893 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znh7x\" (UniqueName: \"kubernetes.io/projected/8def7913-f8e7-44e8-a65f-be8a30fff816-kube-api-access-znh7x\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:39:23.498924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:23.498922 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8def7913-f8e7-44e8-a65f-be8a30fff816-webhook-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:39:24.240059 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.240013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:39:24.323580 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.323554 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74bd8b57c-9whxb"] Jul 22 11:39:24.323926 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.323914 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8def7913-f8e7-44e8-a65f-be8a30fff816" containerName="manager" Jul 22 11:39:24.323997 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.323928 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8def7913-f8e7-44e8-a65f-be8a30fff816" containerName="manager" Jul 22 11:39:24.323997 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.323994 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8def7913-f8e7-44e8-a65f-be8a30fff816" containerName="manager" Jul 22 11:39:24.327962 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.327935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.330970 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.330938 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 11:39:24.331087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.330979 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 11:39:24.331087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.331025 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52s5k\"" Jul 22 11:39:24.338856 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.338833 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-9whxb"] Jul 22 11:39:24.350650 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.350622 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8def7913-f8e7-44e8-a65f-be8a30fff816" path="/var/lib/kubelet/pods/8def7913-f8e7-44e8-a65f-be8a30fff816/volumes" Jul 22 11:39:24.407592 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.407566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa60025-11c9-437d-92b0-c982b1a21f37-webhook-cert\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.407707 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.407596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9n7k\" (UniqueName: \"kubernetes.io/projected/caa60025-11c9-437d-92b0-c982b1a21f37-kube-api-access-d9n7k\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.508911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.508820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa60025-11c9-437d-92b0-c982b1a21f37-webhook-cert\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.508911 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.508873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9n7k\" (UniqueName: \"kubernetes.io/projected/caa60025-11c9-437d-92b0-c982b1a21f37-kube-api-access-d9n7k\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.511474 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.511450 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/caa60025-11c9-437d-92b0-c982b1a21f37-webhook-cert\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.521711 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.521692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9n7k\" (UniqueName: \"kubernetes.io/projected/caa60025-11c9-437d-92b0-c982b1a21f37-kube-api-access-d9n7k\") pod \"maas-controller-74bd8b57c-9whxb\" (UID: \"caa60025-11c9-437d-92b0-c982b1a21f37\") " pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.638010 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.637982 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:24.776416 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:24.776388 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74bd8b57c-9whxb"] Jul 22 11:39:24.778145 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:39:24.778114 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa60025_11c9_437d_92b0_c982b1a21f37.slice/crio-c4808eb3ef7342ac698b2c333089f74e998435b0d208408b0b955a8ed8f3ebd2 WatchSource:0}: Error finding container c4808eb3ef7342ac698b2c333089f74e998435b0d208408b0b955a8ed8f3ebd2: Status 404 returned error can't find the container with id c4808eb3ef7342ac698b2c333089f74e998435b0d208408b0b955a8ed8f3ebd2 Jul 22 11:39:25.424794 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:25.424697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-9whxb" event={"ID":"caa60025-11c9-437d-92b0-c982b1a21f37","Type":"ContainerStarted","Data":"7671b0be8cdfcdbee29b2550068d44db4956f635f9879466c6502295814b8b96"} Jul 22 11:39:25.424794 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:25.424751 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74bd8b57c-9whxb" event={"ID":"caa60025-11c9-437d-92b0-c982b1a21f37","Type":"ContainerStarted","Data":"c4808eb3ef7342ac698b2c333089f74e998435b0d208408b0b955a8ed8f3ebd2"} Jul 22 11:39:25.425364 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:25.424795 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:39:25.447219 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:25.447176 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74bd8b57c-9whxb" podStartSLOduration=1.129146984 podStartE2EDuration="1.447160826s" totalImagesPullingTime="318.013842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:39:25.445696566 +0000 UTC m=+935.607239262" watchObservedRunningTime="2026-07-22 11:39:25.447160826 +0000 UTC m=+935.608703521" Jul 22 11:39:31.835423 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:31.835386 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:39:36.432673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:39:36.432630 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74bd8b57c-9whxb" Jul 22 11:40:00.252444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:00.252403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:02.236489 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:02.236451 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:06.930664 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:06.930617 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:07.002282 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:07.002243 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:07.080238 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:07.080196 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:07.384579 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:07.384540 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:07.452927 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:07.452896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:13.553216 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:13.553098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:40:13.654027 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:40:13.653984 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:42:30.547821 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:42:30.547775 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:43:50.286345 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:43:50.286313 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:43:50.287924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:43:50.287903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:43:54.095997 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:43:54.095941 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:46:48.285679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:46:48.285645 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:46:49.678659 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:46:49.678624 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:47:24.069059 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:47:24.068986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:47:54.250136 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:47:54.250103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:47:59.320386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:47:59.320349 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:48:36.890993 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:48:36.890936 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:48:38.670450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:48:38.670416 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:48:50.312606 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:48:50.312578 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:48:50.315251 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:48:50.315227 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:50:39.489695 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:50:39.489644 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:15.314071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:51:15.313977 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:15.414944 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:51:15.414904 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:51:48.964967 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:51:48.964905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:10.521389 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:10.521287 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:13.623687 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:13.623656 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:13.723716 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:13.723667 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:14.113413 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:14.113361 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:20.419277 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.419242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:20.500479 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.500449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:20.677374 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.677258 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:20.986381 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.986301 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:52:20.989784 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.989765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:20.992736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:20.992717 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8faa1d52-openshift-default-dockercfg-4qqpv\"" Jul 22 11:52:21.012000 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.011939 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:21.013897 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.013869 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:52:21.148091 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148091 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148288 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4dd\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148326 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.148558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.148408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.194673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.194632 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:21.249134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249134 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4dd\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.249807 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.249622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.250080 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.250023 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.250202 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.250100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.250335 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.250313 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.250484 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.250464 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.251753 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.251735 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.252232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.252214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.265710 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.265666 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.267008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.266983 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4dd\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd\") pod \"e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.299672 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.299637 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:21.444686 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.444575 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:52:21.446907 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:52:21.446871 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e39ab4d_dec4_4f21_8f8b_bef60fed4c9a.slice/crio-8e42f7a92b70176e9963457203e377ba946f571dd30f318a9e97874ecb211189 WatchSource:0}: Error finding container 8e42f7a92b70176e9963457203e377ba946f571dd30f318a9e97874ecb211189: Status 404 returned error can't find the container with id 8e42f7a92b70176e9963457203e377ba946f571dd30f318a9e97874ecb211189 Jul 22 11:52:21.449079 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.449045 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:21.449192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.449126 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:21.449192 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:21.449168 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:22.079501 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:22.079461 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerStarted","Data":"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3"} Jul 22 11:52:22.079682 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:22.079528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerStarted","Data":"8e42f7a92b70176e9963457203e377ba946f571dd30f318a9e97874ecb211189"} Jul 22 11:52:22.101501 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:22.101445 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podStartSLOduration=2.10142898 podStartE2EDuration="2.10142898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:52:22.099366191 +0000 UTC m=+1712.260908888" watchObservedRunningTime="2026-07-22 11:52:22.10142898 +0000 UTC m=+1712.262971675" Jul 22 11:52:22.300847 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:22.300804 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:22.306073 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:22.306047 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:23.084112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.084082 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:23.085223 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.085199 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:23.556308 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.556277 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:52:23.559916 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.559890 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.562146 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.562120 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8faa1d52-serving-cert\"" Jul 22 11:52:23.576907 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.576284 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:52:23.674115 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.674079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.674115 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.674117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8mw\" (UniqueName: \"kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.674351 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.674182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.775269 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.775223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.775455 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.775332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.775455 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.775359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8mw\" (UniqueName: \"kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.775880 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.775857 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.778185 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.778156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.787220 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.787193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8mw\" (UniqueName: \"kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw\") pod \"maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.826936 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.826844 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:23.880928 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.880876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:23.929071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:23.929036 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:24.023737 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:24.023703 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:52:24.027577 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:52:24.027546 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f82573_7cae_4a6b_b670_37939ca2080c.slice/crio-7a4cb6c2bec10b552a64c2d1803275f2afc5c04b4055d0e89f3deaafe4135121 WatchSource:0}: Error finding container 7a4cb6c2bec10b552a64c2d1803275f2afc5c04b4055d0e89f3deaafe4135121: Status 404 returned error can't find the container with id 7a4cb6c2bec10b552a64c2d1803275f2afc5c04b4055d0e89f3deaafe4135121 Jul 22 11:52:24.029376 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:24.029358 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:52:24.089787 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:24.089690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" event={"ID":"65f82573-7cae-4a6b-b670-37939ca2080c","Type":"ContainerStarted","Data":"7a4cb6c2bec10b552a64c2d1803275f2afc5c04b4055d0e89f3deaafe4135121"} Jul 22 11:52:24.212491 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:24.212428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:24.306054 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:24.306009 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:25.096495 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:25.096446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" event={"ID":"65f82573-7cae-4a6b-b670-37939ca2080c","Type":"ContainerStarted","Data":"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616"} Jul 22 11:52:25.096935 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:25.096727 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:25.115103 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:25.115044 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" podStartSLOduration=1.522081198 podStartE2EDuration="2.115022114s" totalImagesPullingTime="592.940916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:52:25.11240919 +0000 UTC m=+1715.273951876" watchObservedRunningTime="2026-07-22 11:52:25.115022114 +0000 UTC m=+1715.276564811" Jul 22 11:52:31.107998 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:31.107937 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:52:34.520853 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:34.520766 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:34.622131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:34.622082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:38.085903 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.085863 2569 patch_prober.go:28] interesting pod/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 11:52:38.086365 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.085920 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 11:52:38.165421 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.165387 2569 generic.go:356] "Generic (PLEG): container finished" podID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerID="714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3" exitCode=0 Jul 22 11:52:38.165587 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.165465 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerDied","Data":"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3"} Jul 22 11:52:38.165899 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.165881 2569 scope.go:117] "RemoveContainer" containerID="714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3" Jul 22 11:52:38.166969 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.166917 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:38.167095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.167015 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:38.167095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:38.167048 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:52:39.172701 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:39.172668 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerStarted","Data":"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf"} Jul 22 11:52:39.300729 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:39.300692 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:39.300999 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:39.300975 2569 patch_prober.go:28] interesting pod/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 11:52:39.301055 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:39.301028 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 11:52:40.300190 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:40.300157 2569 patch_prober.go:28] interesting pod/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 11:52:40.300684 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:40.300218 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 11:52:41.300543 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:41.300512 2569 patch_prober.go:28] interesting pod/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 11:52:41.300946 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:41.300564 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 11:52:42.300450 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:42.300413 2569 patch_prober.go:28] interesting pod/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 11:52:42.300619 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:42.300467 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 11:52:43.304533 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:43.304498 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:43.305054 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:43.304884 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:43.305606 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:43.305585 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:52:59.715079 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:59.715044 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:52:59.715498 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:59.715265 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" podUID="65f82573-7cae-4a6b-b670-37939ca2080c" containerName="maas-api" containerID="cri-o://426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616" gracePeriod=30 Jul 22 11:52:59.761980 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:59.761939 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:52:59.970927 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:52:59.970869 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:53:00.097531 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.097510 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls\") pod \"65f82573-7cae-4a6b-b670-37939ca2080c\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " Jul 22 11:53:00.097680 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.097606 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq8mw\" (UniqueName: \"kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw\") pod \"65f82573-7cae-4a6b-b670-37939ca2080c\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " Jul 22 11:53:00.097680 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.097633 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca\") pod \"65f82573-7cae-4a6b-b670-37939ca2080c\" (UID: \"65f82573-7cae-4a6b-b670-37939ca2080c\") " Jul 22 11:53:00.098425 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.098385 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65f82573-7cae-4a6b-b670-37939ca2080c" (UID: "65f82573-7cae-4a6b-b670-37939ca2080c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:53:00.105725 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.105640 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65f82573-7cae-4a6b-b670-37939ca2080c" (UID: "65f82573-7cae-4a6b-b670-37939ca2080c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:53:00.106882 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.106854 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw" (OuterVolumeSpecName: "kube-api-access-tq8mw") pod "65f82573-7cae-4a6b-b670-37939ca2080c" (UID: "65f82573-7cae-4a6b-b670-37939ca2080c"). InnerVolumeSpecName "kube-api-access-tq8mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:53:00.199301 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.199276 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f82573-7cae-4a6b-b670-37939ca2080c-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:00.199385 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.199317 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f82573-7cae-4a6b-b670-37939ca2080c-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:00.199385 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.199327 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8mw\" (UniqueName: \"kubernetes.io/projected/65f82573-7cae-4a6b-b670-37939ca2080c-kube-api-access-tq8mw\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:00.285243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.285188 2569 generic.go:356] "Generic (PLEG): container finished" podID="65f82573-7cae-4a6b-b670-37939ca2080c" containerID="426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616" exitCode=0 Jul 22 11:53:00.285243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.285243 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" Jul 22 11:53:00.285405 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.285287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" event={"ID":"65f82573-7cae-4a6b-b670-37939ca2080c","Type":"ContainerDied","Data":"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616"} Jul 22 11:53:00.285405 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.285382 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq" event={"ID":"65f82573-7cae-4a6b-b670-37939ca2080c","Type":"ContainerDied","Data":"7a4cb6c2bec10b552a64c2d1803275f2afc5c04b4055d0e89f3deaafe4135121"} Jul 22 11:53:00.285570 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.285416 2569 scope.go:117] "RemoveContainer" containerID="426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616" Jul 22 11:53:00.294601 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.294585 2569 scope.go:117] "RemoveContainer" containerID="426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616" Jul 22 11:53:00.294847 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:53:00.294828 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616\": container with ID starting with 426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616 not found: ID does not exist" containerID="426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616" Jul 22 11:53:00.294906 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.294859 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616"} err="failed to get container status \"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616\": rpc error: code = NotFound desc = could not find container \"426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616\": container with ID starting with 426744fc83d73bf57891237f93da20a7330de892b9fdfccebf8e0c26cd5aa616 not found: ID does not exist" Jul 22 11:53:00.307810 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.307790 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:53:00.312236 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.312217 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8faa1d52-5fbd4dd88b-zb4nq"] Jul 22 11:53:00.349828 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:00.349802 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f82573-7cae-4a6b-b670-37939ca2080c" path="/var/lib/kubelet/pods/65f82573-7cae-4a6b-b670-37939ca2080c/volumes" Jul 22 11:53:04.385311 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.385279 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:53:04.385783 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.385461 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" containerID="cri-o://474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf" gracePeriod=30 Jul 22 11:53:04.418257 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.418235 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:04.491096 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.491063 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:04.648164 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.648094 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:04.900888 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.900817 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:04.995313 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:04.995283 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:05.124151 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:05.124126 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:05.461835 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:05.461799 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:05.662701 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:05.662670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:09.631725 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.631704 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:53:09.776460 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776384 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn4dd\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776460 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776429 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776460 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776450 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776487 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776526 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776553 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776568 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776598 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776629 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data\") pod \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\" (UID: \"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a\") " Jul 22 11:53:09.776720 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776677 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:53:09.777133 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776830 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:53:09.777133 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.776909 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:53:09.777133 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.777085 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:53:09.777464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.777157 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-certs\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.777464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.777180 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istiod-ca-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.777464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.777194 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-credential-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.777631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.777548 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:53:09.779206 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.779182 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 11:53:09.779310 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.779292 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:53:09.779409 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.779385 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:53:09.779448 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.779398 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd" (OuterVolumeSpecName: "kube-api-access-pn4dd") pod "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" (UID: "9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a"). InnerVolumeSpecName "kube-api-access-pn4dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:53:09.878324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878301 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-podinfo\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.878324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878321 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-data\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.878463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878331 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4dd\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-kube-api-access-pn4dd\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.878463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878341 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-token\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.878463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878349 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-istio-envoy\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:09.878463 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:09.878357 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a-workload-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:10.346508 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.346478 2569 generic.go:356] "Generic (PLEG): container finished" podID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerID="474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf" exitCode=0 Jul 22 11:53:10.346661 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.346590 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" Jul 22 11:53:10.350249 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.350221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerDied","Data":"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf"} Jul 22 11:53:10.350398 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.350265 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw" event={"ID":"9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a","Type":"ContainerDied","Data":"8e42f7a92b70176e9963457203e377ba946f571dd30f318a9e97874ecb211189"} Jul 22 11:53:10.350398 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.350289 2569 scope.go:117] "RemoveContainer" containerID="474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf" Jul 22 11:53:10.361131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.361114 2569 scope.go:117] "RemoveContainer" containerID="714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3" Jul 22 11:53:10.369351 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.369332 2569 scope.go:117] "RemoveContainer" containerID="474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf" Jul 22 11:53:10.369615 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:53:10.369597 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf\": container with ID starting with 474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf not found: ID does not exist" containerID="474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf" Jul 22 11:53:10.369679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.369622 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf"} err="failed to get container status \"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf\": rpc error: code = NotFound desc = could not find container \"474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf\": container with ID starting with 474ebb839ae12360672b44f8121db263d76bd75695969e14683e332c91e836bf not found: ID does not exist" Jul 22 11:53:10.369679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.369638 2569 scope.go:117] "RemoveContainer" containerID="714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3" Jul 22 11:53:10.369893 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:53:10.369875 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3\": container with ID starting with 714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3 not found: ID does not exist" containerID="714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3" Jul 22 11:53:10.369940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.369897 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3"} err="failed to get container status \"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3\": rpc error: code = NotFound desc = could not find container \"714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3\": container with ID starting with 714a61de4d79e66022f50b2109a35bab10d8a9b0a85f07600cc68332a80138f3 not found: ID does not exist" Jul 22 11:53:10.374229 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.374205 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:53:10.381229 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:10.381210 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8faa1d52-openshift-default-d5bc5f5f-pv6dw"] Jul 22 11:53:11.038878 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.038845 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:11.128697 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.128661 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:11.129099 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129084 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f82573-7cae-4a6b-b670-37939ca2080c" containerName="maas-api" Jul 22 11:53:11.129150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129107 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f82573-7cae-4a6b-b670-37939ca2080c" containerName="maas-api" Jul 22 11:53:11.129150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129123 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:11.129150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129129 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:11.129150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129134 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:11.129150 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129139 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:11.129300 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129202 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:11.129300 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.129210 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f82573-7cae-4a6b-b670-37939ca2080c" containerName="maas-api" Jul 22 11:53:11.133580 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.133564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.135537 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.135515 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d8c0f9e9-serving-cert\"" Jul 22 11:53:11.151167 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.147648 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:11.168245 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.168220 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:11.290908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.290834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.290908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.290869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.290908 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.290897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8h9w\" (UniqueName: \"kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.352570 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.352542 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:11.392185 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.392163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.392294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.392192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.392294 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.392221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8h9w\" (UniqueName: \"kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.392765 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.392747 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.394754 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.394732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.404704 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.404678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8h9w\" (UniqueName: \"kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w\") pod \"maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.447649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.447624 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:11.797824 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.797763 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:11.798042 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.798024 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:11.800166 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:53:11.800134 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e55da_e0c4_43b2_ab10_258ce4a523c3.slice/crio-8bc2c4d8bbd8caaab824d7d02ed5d866c89ce42c6c1b085998775a26877148f4 WatchSource:0}: Error finding container 8bc2c4d8bbd8caaab824d7d02ed5d866c89ce42c6c1b085998775a26877148f4: Status 404 returned error can't find the container with id 8bc2c4d8bbd8caaab824d7d02ed5d866c89ce42c6c1b085998775a26877148f4 Jul 22 11:53:11.896221 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:11.896197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:12.350131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:12.350098 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" path="/var/lib/kubelet/pods/9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a/volumes" Jul 22 11:53:12.357216 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:12.357188 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" event={"ID":"652e55da-e0c4-43b2-ab10-258ce4a523c3","Type":"ContainerStarted","Data":"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb"} Jul 22 11:53:12.357331 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:12.357234 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:12.357331 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:12.357249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" event={"ID":"652e55da-e0c4-43b2-ab10-258ce4a523c3","Type":"ContainerStarted","Data":"8bc2c4d8bbd8caaab824d7d02ed5d866c89ce42c6c1b085998775a26877148f4"} Jul 22 11:53:12.378827 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:12.378775 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" podStartSLOduration=0.963873161 podStartE2EDuration="1.378756558s" totalImagesPullingTime="414.883397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:53:12.377050218 +0000 UTC m=+1762.538592945" watchObservedRunningTime="2026-07-22 11:53:12.378756558 +0000 UTC m=+1762.540299254" Jul 22 11:53:18.367768 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:18.367734 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:46.750267 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:46.750235 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:46.750694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:46.750437 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" podUID="652e55da-e0c4-43b2-ab10-258ce4a523c3" containerName="maas-api" containerID="cri-o://a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb" gracePeriod=30 Jul 22 11:53:46.824825 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:46.824798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:46.998755 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:46.998729 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:47.075961 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.075927 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8h9w\" (UniqueName: \"kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w\") pod \"652e55da-e0c4-43b2-ab10-258ce4a523c3\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " Jul 22 11:53:47.076103 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.075995 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls\") pod \"652e55da-e0c4-43b2-ab10-258ce4a523c3\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " Jul 22 11:53:47.076103 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.076054 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca\") pod \"652e55da-e0c4-43b2-ab10-258ce4a523c3\" (UID: \"652e55da-e0c4-43b2-ab10-258ce4a523c3\") " Jul 22 11:53:47.076408 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.076370 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "652e55da-e0c4-43b2-ab10-258ce4a523c3" (UID: "652e55da-e0c4-43b2-ab10-258ce4a523c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:53:47.078097 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.078072 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w" (OuterVolumeSpecName: "kube-api-access-b8h9w") pod "652e55da-e0c4-43b2-ab10-258ce4a523c3" (UID: "652e55da-e0c4-43b2-ab10-258ce4a523c3"). InnerVolumeSpecName "kube-api-access-b8h9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:53:47.078184 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.078115 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "652e55da-e0c4-43b2-ab10-258ce4a523c3" (UID: "652e55da-e0c4-43b2-ab10-258ce4a523c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:53:47.177182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.177149 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652e55da-e0c4-43b2-ab10-258ce4a523c3-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:47.177182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.177179 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8h9w\" (UniqueName: \"kubernetes.io/projected/652e55da-e0c4-43b2-ab10-258ce4a523c3-kube-api-access-b8h9w\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:47.177182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.177190 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652e55da-e0c4-43b2-ab10-258ce4a523c3-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:53:47.530678 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.530608 2569 generic.go:356] "Generic (PLEG): container finished" podID="652e55da-e0c4-43b2-ab10-258ce4a523c3" containerID="a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb" exitCode=0 Jul 22 11:53:47.530678 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.530658 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" Jul 22 11:53:47.530678 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.530668 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" event={"ID":"652e55da-e0c4-43b2-ab10-258ce4a523c3","Type":"ContainerDied","Data":"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb"} Jul 22 11:53:47.530895 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.530708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj" event={"ID":"652e55da-e0c4-43b2-ab10-258ce4a523c3","Type":"ContainerDied","Data":"8bc2c4d8bbd8caaab824d7d02ed5d866c89ce42c6c1b085998775a26877148f4"} Jul 22 11:53:47.530895 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.530729 2569 scope.go:117] "RemoveContainer" containerID="a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb" Jul 22 11:53:47.540649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.540625 2569 scope.go:117] "RemoveContainer" containerID="a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb" Jul 22 11:53:47.540946 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:53:47.540925 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb\": container with ID starting with a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb not found: ID does not exist" containerID="a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb" Jul 22 11:53:47.541025 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.540985 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb"} err="failed to get container status \"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb\": rpc error: code = NotFound desc = could not find container \"a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb\": container with ID starting with a4dfad1111c3b6431e6cc7561f7d188d085f0b5258c0ebc7e5cde4c98d64f6cb not found: ID does not exist" Jul 22 11:53:47.555689 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.555663 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:47.566731 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:47.566704 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8c0f9e9-5b5bc5fc9f-2tdtj"] Jul 22 11:53:48.350896 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:48.350857 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652e55da-e0c4-43b2-ab10-258ce4a523c3" path="/var/lib/kubelet/pods/652e55da-e0c4-43b2-ab10-258ce4a523c3/volumes" Jul 22 11:53:50.339971 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:50.339932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:53:50.342705 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:50.342685 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:53:51.296240 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:51.296204 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:51.534633 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:51.534590 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:51.740050 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:51.740022 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:51.979261 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:51.979230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:52.272319 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.272284 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:53:52.272672 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.272660 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652e55da-e0c4-43b2-ab10-258ce4a523c3" containerName="maas-api" Jul 22 11:53:52.272717 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.272674 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="652e55da-e0c4-43b2-ab10-258ce4a523c3" containerName="maas-api" Jul 22 11:53:52.272751 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.272737 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="652e55da-e0c4-43b2-ab10-258ce4a523c3" containerName="maas-api" Jul 22 11:53:52.272751 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.272746 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e39ab4d-dec4-4f21-8f8b-bef60fed4c9a" containerName="istio-proxy" Jul 22 11:53:52.277001 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.276986 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.279086 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.279066 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-beed8e14-openshift-default-dockercfg-2n6p5\"" Jul 22 11:53:52.290422 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.290400 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:53:52.317794 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.317893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317803 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km4f\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.317893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.317893 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.318019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.318019 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.317985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.318087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.318067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.318120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.318085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.318120 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.318107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.368483 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.368449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:52.418679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.418679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418689 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.418891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.418891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.418891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.418891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.418855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419139 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km4f\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419139 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419257 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419257 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419173 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419257 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419185 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419403 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419789 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419767 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.419898 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.419836 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.421292 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.421269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.421574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.421558 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.428595 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.428569 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km4f\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.430368 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.430351 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token\") pod \"e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.553748 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.553711 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:52.587224 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.587184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:52.729290 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.729264 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:53:52.731153 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:53:52.731124 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a62f361_f615_42a5_81a5_1acf7682a46e.slice/crio-bc04b0448823d077b518a589b6346a00c5593babbd9f4cf09addc73611d9817e WatchSource:0}: Error finding container bc04b0448823d077b518a589b6346a00c5593babbd9f4cf09addc73611d9817e: Status 404 returned error can't find the container with id bc04b0448823d077b518a589b6346a00c5593babbd9f4cf09addc73611d9817e Jul 22 11:53:52.733188 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.733155 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:53:52.733284 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.733232 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:53:52.733284 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:52.733275 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:53:53.563016 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:53.562985 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerStarted","Data":"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167"} Jul 22 11:53:53.563365 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:53.563034 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerStarted","Data":"bc04b0448823d077b518a589b6346a00c5593babbd9f4cf09addc73611d9817e"} Jul 22 11:53:53.587677 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:53.587651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:53.592392 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:53.592337 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podStartSLOduration=1.592322078 podStartE2EDuration="1.592322078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:53:53.590262769 +0000 UTC m=+1803.751805465" watchObservedRunningTime="2026-07-22 11:53:53.592322078 +0000 UTC m=+1803.753864772" Jul 22 11:53:53.592997 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:53.592979 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:54.567212 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:54.567187 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:54.567998 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:54.567977 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:53:55.124527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.124496 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:55.154657 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.154620 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:53:55.165295 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.165271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.172909 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.172885 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-beed8e14-serving-cert\"" Jul 22 11:53:55.176569 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.176534 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:53:55.243325 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.243304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.243430 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.243367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb55\" (UniqueName: \"kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.243430 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.243395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.343980 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.343927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.344141 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.344067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb55\" (UniqueName: \"kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.344141 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.344098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.344666 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.344643 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.346464 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.346446 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.355830 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.355807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb55\" (UniqueName: \"kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55\") pod \"maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.458035 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.457947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:55.476649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.476631 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:55.645936 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:55.645913 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:53:55.648004 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:53:55.647980 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1409546_0ab3_4a45_81b4_9d2e6b56189a.slice/crio-15a3d74845f671957f84078d6e39bba5829e3de699bffca88ef622b97a59a4fa WatchSource:0}: Error finding container 15a3d74845f671957f84078d6e39bba5829e3de699bffca88ef622b97a59a4fa: Status 404 returned error can't find the container with id 15a3d74845f671957f84078d6e39bba5829e3de699bffca88ef622b97a59a4fa Jul 22 11:53:56.155502 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.155472 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:56.274167 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.274123 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:53:56.581008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.580974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" event={"ID":"f1409546-0ab3-4a45-81b4-9d2e6b56189a","Type":"ContainerStarted","Data":"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c"} Jul 22 11:53:56.581228 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.581206 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:53:56.581298 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.581265 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" event={"ID":"f1409546-0ab3-4a45-81b4-9d2e6b56189a","Type":"ContainerStarted","Data":"15a3d74845f671957f84078d6e39bba5829e3de699bffca88ef622b97a59a4fa"} Jul 22 11:53:56.601871 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:53:56.601825 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" podStartSLOduration=1.179660437 podStartE2EDuration="1.601810919s" totalImagesPullingTime="422.150482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:53:56.600785083 +0000 UTC m=+1806.762327778" watchObservedRunningTime="2026-07-22 11:53:56.601810919 +0000 UTC m=+1806.763353616" Jul 22 11:54:02.597527 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:02.597496 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:54:09.568658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.568627 2569 patch_prober.go:28] interesting pod/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 11:54:09.569084 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.568678 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 11:54:09.651472 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.651445 2569 generic.go:356] "Generic (PLEG): container finished" podID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerID="d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167" exitCode=0 Jul 22 11:54:09.651608 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.651494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerDied","Data":"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167"} Jul 22 11:54:09.651848 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.651836 2569 scope.go:117] "RemoveContainer" containerID="d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167" Jul 22 11:54:09.652931 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.652873 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:54:09.653110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.653046 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:54:09.653162 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:09.653113 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 11:54:10.657836 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:10.657802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerStarted","Data":"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947"} Jul 22 11:54:11.588119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:11.588082 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:11.588324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:11.588199 2569 patch_prober.go:28] interesting pod/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 11:54:11.588324 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:11.588241 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 11:54:12.587649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:12.587605 2569 patch_prober.go:28] interesting pod/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 11:54:12.588159 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:12.587669 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 11:54:13.588332 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:13.588296 2569 patch_prober.go:28] interesting pod/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 11:54:13.588867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:13.588837 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 11:54:14.591402 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:14.591370 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:14.591781 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:14.591629 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:14.592338 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:14.592321 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:31.488270 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.488234 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:54:31.488764 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.488461 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" podUID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" containerName="maas-api" containerID="cri-o://cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c" gracePeriod=30 Jul 22 11:54:31.527679 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.527654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:31.747504 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.747435 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:54:31.751773 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.751750 2569 generic.go:356] "Generic (PLEG): container finished" podID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" containerID="cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c" exitCode=0 Jul 22 11:54:31.751904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.751792 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" Jul 22 11:54:31.751904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.751837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" event={"ID":"f1409546-0ab3-4a45-81b4-9d2e6b56189a","Type":"ContainerDied","Data":"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c"} Jul 22 11:54:31.751904 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.751885 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw" event={"ID":"f1409546-0ab3-4a45-81b4-9d2e6b56189a","Type":"ContainerDied","Data":"15a3d74845f671957f84078d6e39bba5829e3de699bffca88ef622b97a59a4fa"} Jul 22 11:54:31.752043 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.751907 2569 scope.go:117] "RemoveContainer" containerID="cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c" Jul 22 11:54:31.760119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.760103 2569 scope.go:117] "RemoveContainer" containerID="cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c" Jul 22 11:54:31.760385 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:54:31.760369 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c\": container with ID starting with cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c not found: ID does not exist" containerID="cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c" Jul 22 11:54:31.760444 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.760395 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c"} err="failed to get container status \"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c\": rpc error: code = NotFound desc = could not find container \"cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c\": container with ID starting with cec0972ac7bb47330965ec836c419f195b926c3cf8a454958204921dd87dff8c not found: ID does not exist" Jul 22 11:54:31.851565 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.851537 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwb55\" (UniqueName: \"kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55\") pod \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " Jul 22 11:54:31.851684 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.851615 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca\") pod \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " Jul 22 11:54:31.851744 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.851716 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls\") pod \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\" (UID: \"f1409546-0ab3-4a45-81b4-9d2e6b56189a\") " Jul 22 11:54:31.851993 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.851944 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1409546-0ab3-4a45-81b4-9d2e6b56189a" (UID: "f1409546-0ab3-4a45-81b4-9d2e6b56189a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:54:31.853682 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.853657 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55" (OuterVolumeSpecName: "kube-api-access-dwb55") pod "f1409546-0ab3-4a45-81b4-9d2e6b56189a" (UID: "f1409546-0ab3-4a45-81b4-9d2e6b56189a"). InnerVolumeSpecName "kube-api-access-dwb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:54:31.853847 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.853828 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1409546-0ab3-4a45-81b4-9d2e6b56189a" (UID: "f1409546-0ab3-4a45-81b4-9d2e6b56189a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:54:31.953316 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.953289 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1409546-0ab3-4a45-81b4-9d2e6b56189a-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:31.953316 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.953314 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1409546-0ab3-4a45-81b4-9d2e6b56189a-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:31.953458 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:31.953323 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwb55\" (UniqueName: \"kubernetes.io/projected/f1409546-0ab3-4a45-81b4-9d2e6b56189a-kube-api-access-dwb55\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:32.076322 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:32.076299 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:54:32.082891 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:32.082866 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-beed8e14-7c547fcbf8-wfjgw"] Jul 22 11:54:32.350384 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:32.350322 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" path="/var/lib/kubelet/pods/f1409546-0ab3-4a45-81b4-9d2e6b56189a/volumes" Jul 22 11:54:36.034835 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:36.034796 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:37.402177 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:37.402148 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:54:37.402566 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:37.402352 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" containerID="cri-o://ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947" gracePeriod=30 Jul 22 11:54:37.456202 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:37.456178 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:37.702729 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:37.702648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:37.999106 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:37.999037 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:38.395897 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:38.395871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:38.868066 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:38.868034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:40.423473 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.423439 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:40.510208 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.510180 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:54:40.510560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.510542 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" containerName="maas-api" Jul 22 11:54:40.510560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.510559 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" containerName="maas-api" Jul 22 11:54:40.510762 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.510652 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1409546-0ab3-4a45-81b4-9d2e6b56189a" containerName="maas-api" Jul 22 11:54:40.514969 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.514939 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.517243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.517225 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9906803d-serving-cert\"" Jul 22 11:54:40.528848 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.528828 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:54:40.545986 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.545937 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:40.624524 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.624473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.624672 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.624528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzvh\" (UniqueName: \"kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.624672 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.624587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.725644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.725572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.725644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.725601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzvh\" (UniqueName: \"kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.725644 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.725636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.725914 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:54:40.725713 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-serving-cert: secret "maas-api-e2e-derive-9906803d-serving-cert" not found Jul 22 11:54:40.725914 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:54:40.725770 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls podName:8f13696e-a7de-414e-920b-def20b5f26d6 nodeName:}" failed. No retries permitted until 2026-07-22 11:54:41.225755164 +0000 UTC m=+1851.387297838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls") pod "maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" (UID: "8f13696e-a7de-414e-920b-def20b5f26d6") : secret "maas-api-e2e-derive-9906803d-serving-cert" not found Jul 22 11:54:40.726282 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.726265 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.735693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.735665 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzvh\" (UniqueName: \"kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:40.757628 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:40.757603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:41.230536 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.230505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:41.233109 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.233085 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") pod \"maas-api-e2e-derive-9906803d-64b5c8fd78-s26st\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:41.245087 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.245064 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:41.346363 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.346342 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:41.424386 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.424345 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:41.586633 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.586603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:54:41.588831 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:54:41.588800 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f13696e_a7de_414e_920b_def20b5f26d6.slice/crio-b96d48e89763f77b85c975bc1ba562be67e12f839859a7ed6500f4aca19cc8a1 WatchSource:0}: Error finding container b96d48e89763f77b85c975bc1ba562be67e12f839859a7ed6500f4aca19cc8a1: Status 404 returned error can't find the container with id b96d48e89763f77b85c975bc1ba562be67e12f839859a7ed6500f4aca19cc8a1 Jul 22 11:54:41.798756 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:41.798727 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" event={"ID":"8f13696e-a7de-414e-920b-def20b5f26d6","Type":"ContainerStarted","Data":"b96d48e89763f77b85c975bc1ba562be67e12f839859a7ed6500f4aca19cc8a1"} Jul 22 11:54:42.662568 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.662547 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:42.744071 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744046 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744195 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744080 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744195 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744124 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744195 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744157 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5km4f\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744195 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744173 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744219 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744254 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744287 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744299 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:54:42.744620 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744307 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert\") pod \"9a62f361-f615-42a5-81a5-1acf7682a46e\" (UID: \"9a62f361-f615-42a5-81a5-1acf7682a46e\") " Jul 22 11:54:42.744620 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744510 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:54:42.744620 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.744575 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:54:42.745162 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745137 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:54:42.745414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745383 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:54:42.745572 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745551 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-certs\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.745637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745578 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-data\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.745637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745592 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-credential-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.745637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745606 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a62f361-f615-42a5-81a5-1acf7682a46e-istiod-ca-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.745637 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.745619 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-workload-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.746697 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.746663 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:54:42.746786 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.746706 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f" (OuterVolumeSpecName: "kube-api-access-5km4f") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "kube-api-access-5km4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:54:42.746786 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.746736 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:54:42.746877 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.746851 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a62f361-f615-42a5-81a5-1acf7682a46e" (UID: "9a62f361-f615-42a5-81a5-1acf7682a46e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 11:54:42.805602 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.805578 2569 generic.go:356] "Generic (PLEG): container finished" podID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerID="ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947" exitCode=0 Jul 22 11:54:42.805694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.805633 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" Jul 22 11:54:42.805694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.805633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerDied","Data":"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947"} Jul 22 11:54:42.805694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.805689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9" event={"ID":"9a62f361-f615-42a5-81a5-1acf7682a46e","Type":"ContainerDied","Data":"bc04b0448823d077b518a589b6346a00c5593babbd9f4cf09addc73611d9817e"} Jul 22 11:54:42.805847 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.805719 2569 scope.go:117] "RemoveContainer" containerID="ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947" Jul 22 11:54:42.807662 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.807631 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" event={"ID":"8f13696e-a7de-414e-920b-def20b5f26d6","Type":"ContainerStarted","Data":"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2"} Jul 22 11:54:42.807764 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.807714 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:54:42.815295 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.815280 2569 scope.go:117] "RemoveContainer" containerID="d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167" Jul 22 11:54:42.822716 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.822700 2569 scope.go:117] "RemoveContainer" containerID="ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947" Jul 22 11:54:42.822949 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:54:42.822927 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947\": container with ID starting with ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947 not found: ID does not exist" containerID="ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947" Jul 22 11:54:42.823093 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.823051 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947"} err="failed to get container status \"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947\": rpc error: code = NotFound desc = could not find container \"ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947\": container with ID starting with ec1bda2e38bfb395ceabf7b20045305c0e294424052fdf28282a6e50a2505947 not found: ID does not exist" Jul 22 11:54:42.823093 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.823068 2569 scope.go:117] "RemoveContainer" containerID="d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167" Jul 22 11:54:42.823319 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:54:42.823302 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167\": container with ID starting with d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167 not found: ID does not exist" containerID="d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167" Jul 22 11:54:42.823361 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.823325 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167"} err="failed to get container status \"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167\": rpc error: code = NotFound desc = could not find container \"d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167\": container with ID starting with d63fcd9375227e26a5a57a95f685c2975b8adb6116c0dd2cf65cd89fcc05d167 not found: ID does not exist" Jul 22 11:54:42.837545 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.837508 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" podStartSLOduration=2.223630898 podStartE2EDuration="2.837495272s" totalImagesPullingTime="613.864374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:42.836247215 +0000 UTC m=+1852.997789943" watchObservedRunningTime="2026-07-22 11:54:42.837495272 +0000 UTC m=+1852.999038020" Jul 22 11:54:42.846631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.846606 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-podinfo\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.846631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.846625 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-token\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.846631 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.846634 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5km4f\" (UniqueName: \"kubernetes.io/projected/9a62f361-f615-42a5-81a5-1acf7682a46e-kube-api-access-5km4f\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.846784 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.846643 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a62f361-f615-42a5-81a5-1acf7682a46e-istio-envoy\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:54:42.856768 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.856747 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:54:42.862569 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:42.862551 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-beed8e14-openshift-default-6cbf679f9f-8gfq9"] Jul 22 11:54:44.350452 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:44.350411 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" path="/var/lib/kubelet/pods/9a62f361-f615-42a5-81a5-1acf7682a46e/volumes" Jul 22 11:54:47.104036 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:47.104004 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:47.314921 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:47.314888 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:54:48.819485 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:54:48.819460 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:55:02.424095 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:02.424061 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:03.119937 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:03.119900 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:15.937894 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:15.937807 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:55:15.938458 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:15.938113 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" podUID="8f13696e-a7de-414e-920b-def20b5f26d6" containerName="maas-api" containerID="cri-o://1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2" gracePeriod=30 Jul 22 11:55:15.974808 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:15.974783 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:16.187273 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.187250 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:55:16.310960 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.310934 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjzvh\" (UniqueName: \"kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh\") pod \"8f13696e-a7de-414e-920b-def20b5f26d6\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " Jul 22 11:55:16.311119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.311007 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") pod \"8f13696e-a7de-414e-920b-def20b5f26d6\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " Jul 22 11:55:16.311119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.311078 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca\") pod \"8f13696e-a7de-414e-920b-def20b5f26d6\" (UID: \"8f13696e-a7de-414e-920b-def20b5f26d6\") " Jul 22 11:55:16.311431 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.311406 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f13696e-a7de-414e-920b-def20b5f26d6" (UID: "8f13696e-a7de-414e-920b-def20b5f26d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:55:16.313175 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.313149 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh" (OuterVolumeSpecName: "kube-api-access-pjzvh") pod "8f13696e-a7de-414e-920b-def20b5f26d6" (UID: "8f13696e-a7de-414e-920b-def20b5f26d6"). InnerVolumeSpecName "kube-api-access-pjzvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:55:16.313283 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.313245 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f13696e-a7de-414e-920b-def20b5f26d6" (UID: "8f13696e-a7de-414e-920b-def20b5f26d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:55:16.412126 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.412096 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzvh\" (UniqueName: \"kubernetes.io/projected/8f13696e-a7de-414e-920b-def20b5f26d6-kube-api-access-pjzvh\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:55:16.412126 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.412124 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f13696e-a7de-414e-920b-def20b5f26d6-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:55:16.412286 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.412139 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f13696e-a7de-414e-920b-def20b5f26d6-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:55:16.967171 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.967141 2569 generic.go:356] "Generic (PLEG): container finished" podID="8f13696e-a7de-414e-920b-def20b5f26d6" containerID="1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2" exitCode=0 Jul 22 11:55:16.967574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.967180 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" Jul 22 11:55:16.967574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.967191 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" event={"ID":"8f13696e-a7de-414e-920b-def20b5f26d6","Type":"ContainerDied","Data":"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2"} Jul 22 11:55:16.967574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.967238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st" event={"ID":"8f13696e-a7de-414e-920b-def20b5f26d6","Type":"ContainerDied","Data":"b96d48e89763f77b85c975bc1ba562be67e12f839859a7ed6500f4aca19cc8a1"} Jul 22 11:55:16.967574 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.967261 2569 scope.go:117] "RemoveContainer" containerID="1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2" Jul 22 11:55:16.975658 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.975625 2569 scope.go:117] "RemoveContainer" containerID="1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2" Jul 22 11:55:16.975940 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:55:16.975922 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2\": container with ID starting with 1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2 not found: ID does not exist" containerID="1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2" Jul 22 11:55:16.976037 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.975968 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2"} err="failed to get container status \"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2\": rpc error: code = NotFound desc = could not find container \"1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2\": container with ID starting with 1b347463e9ccf722a5673c72cdc67afaaa7245a231ccb1edc7b438bee0b551f2 not found: ID does not exist" Jul 22 11:55:16.986656 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.986634 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:55:16.992130 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:16.992110 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9906803d-64b5c8fd78-s26st"] Jul 22 11:55:18.351429 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:18.351394 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f13696e-a7de-414e-920b-def20b5f26d6" path="/var/lib/kubelet/pods/8f13696e-a7de-414e-920b-def20b5f26d6/volumes" Jul 22 11:55:20.455924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:20.455882 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:20.639457 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:20.639420 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:20.893740 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:20.893696 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:21.868669 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:21.868642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:21.971334 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:21.971298 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:23.276651 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:23.276615 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:29.849932 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:29.849902 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:30.237838 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:30.237736 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:30.334771 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:30.334732 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:30.666560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:30.666518 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:55:56.767207 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:55:56.767158 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:03.690885 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:03.690853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:03.791484 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:03.791450 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:03.891145 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:03.891118 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:03.991132 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:03.991060 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:24.014126 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:24.014083 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:41.039929 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:41.039847 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:41.141836 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:41.141802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:45.303578 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:45.303543 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:48.699590 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:48.699526 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:51.220849 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:51.220812 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:54.748870 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:54.748830 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:54.862931 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:54.862901 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:56:55.425040 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:56:55.425008 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:02.988180 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:02.988141 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:03.087485 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:03.087429 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:03.318135 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:03.318107 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:03.673045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:03.672940 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:04.021070 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:04.020985 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:07.055446 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055413 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:07.055867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055796 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:57:07.055867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055813 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:57:07.055867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055833 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f13696e-a7de-414e-920b-def20b5f26d6" containerName="maas-api" Jul 22 11:57:07.055867 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055841 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f13696e-a7de-414e-920b-def20b5f26d6" containerName="maas-api" Jul 22 11:57:07.056167 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055947 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f13696e-a7de-414e-920b-def20b5f26d6" containerName="maas-api" Jul 22 11:57:07.056167 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.055977 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:57:07.059100 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.059079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.061243 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.061222 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8709147b-serving-cert\"" Jul 22 11:57:07.073413 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.073390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:07.187347 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.187318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.187481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.187356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.187481 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.187389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62whg\" (UniqueName: \"kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.288080 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.288043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.288244 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.288095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.288244 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.288147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62whg\" (UniqueName: \"kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.288835 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.288814 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.290611 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.290590 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.297110 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.297090 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62whg\" (UniqueName: \"kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg\") pod \"maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.371112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.371063 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:07.504707 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.504680 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:07.507502 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:57:07.507472 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e5352a_d3aa_408c_925b_019a9ce13c0f.slice/crio-792e7ed6bb1ff6d07c61138cc872f9fdc182e41b858bbaa873c2a16ac500b479 WatchSource:0}: Error finding container 792e7ed6bb1ff6d07c61138cc872f9fdc182e41b858bbaa873c2a16ac500b479: Status 404 returned error can't find the container with id 792e7ed6bb1ff6d07c61138cc872f9fdc182e41b858bbaa873c2a16ac500b479 Jul 22 11:57:07.924722 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:07.924680 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:08.191336 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:08.191217 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:08.500448 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:08.500373 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" event={"ID":"e5e5352a-d3aa-408c-925b-019a9ce13c0f","Type":"ContainerStarted","Data":"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db"} Jul 22 11:57:08.500448 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:08.500422 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" event={"ID":"e5e5352a-d3aa-408c-925b-019a9ce13c0f","Type":"ContainerStarted","Data":"792e7ed6bb1ff6d07c61138cc872f9fdc182e41b858bbaa873c2a16ac500b479"} Jul 22 11:57:08.500646 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:08.500522 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:08.520291 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:08.520238 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" podStartSLOduration=1.042619571 podStartE2EDuration="1.520224959s" totalImagesPullingTime="477.605388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:57:08.518972988 +0000 UTC m=+1998.680515690" watchObservedRunningTime="2026-07-22 11:57:08.520224959 +0000 UTC m=+1998.681767686" Jul 22 11:57:11.280693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:11.280643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:11.944673 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:11.944633 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:13.341931 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:13.341882 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:14.509698 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:14.509674 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:18.932558 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:18.932506 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:19.032947 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:19.032912 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:29.848432 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:29.848403 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:29.849018 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:29.848609 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" podUID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" containerName="maas-api" containerID="cri-o://216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db" gracePeriod=30 Jul 22 11:57:30.095076 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.095056 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:30.128264 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.128188 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:30.171476 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.171455 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca\") pod \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " Jul 22 11:57:30.171582 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.171525 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls\") pod \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " Jul 22 11:57:30.171582 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.171566 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62whg\" (UniqueName: \"kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg\") pod \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\" (UID: \"e5e5352a-d3aa-408c-925b-019a9ce13c0f\") " Jul 22 11:57:30.171770 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.171749 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5e5352a-d3aa-408c-925b-019a9ce13c0f" (UID: "e5e5352a-d3aa-408c-925b-019a9ce13c0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:57:30.173625 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.173603 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5e5352a-d3aa-408c-925b-019a9ce13c0f" (UID: "e5e5352a-d3aa-408c-925b-019a9ce13c0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:57:30.173756 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.173737 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg" (OuterVolumeSpecName: "kube-api-access-62whg") pod "e5e5352a-d3aa-408c-925b-019a9ce13c0f" (UID: "e5e5352a-d3aa-408c-925b-019a9ce13c0f"). InnerVolumeSpecName "kube-api-access-62whg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:57:30.272441 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.272419 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e5352a-d3aa-408c-925b-019a9ce13c0f-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:57:30.272441 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.272441 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62whg\" (UniqueName: \"kubernetes.io/projected/e5e5352a-d3aa-408c-925b-019a9ce13c0f-kube-api-access-62whg\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:57:30.272560 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.272451 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5352a-d3aa-408c-925b-019a9ce13c0f-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:57:30.601618 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.601594 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" containerID="216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db" exitCode=0 Jul 22 11:57:30.601736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.601645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" event={"ID":"e5e5352a-d3aa-408c-925b-019a9ce13c0f","Type":"ContainerDied","Data":"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db"} Jul 22 11:57:30.601736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.601650 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" Jul 22 11:57:30.601736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.601685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq" event={"ID":"e5e5352a-d3aa-408c-925b-019a9ce13c0f","Type":"ContainerDied","Data":"792e7ed6bb1ff6d07c61138cc872f9fdc182e41b858bbaa873c2a16ac500b479"} Jul 22 11:57:30.601736 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.601707 2569 scope.go:117] "RemoveContainer" containerID="216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db" Jul 22 11:57:30.610826 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.610808 2569 scope.go:117] "RemoveContainer" containerID="216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db" Jul 22 11:57:30.611108 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:57:30.611089 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db\": container with ID starting with 216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db not found: ID does not exist" containerID="216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db" Jul 22 11:57:30.611181 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.611113 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db"} err="failed to get container status \"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db\": rpc error: code = NotFound desc = could not find container \"216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db\": container with ID starting with 216c1a50b5045a10697b70da9fcd78e83bc37893a9b77a478a85b660419da8db not found: ID does not exist" Jul 22 11:57:30.619927 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.619906 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:30.624004 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:30.623980 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8709147b-6b444dfd7c-kqpbq"] Jul 22 11:57:32.350669 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:32.350624 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" path="/var/lib/kubelet/pods/e5e5352a-d3aa-408c-925b-019a9ce13c0f/volumes" Jul 22 11:57:34.401117 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:34.401085 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:34.806077 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:34.806049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.166727 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.166689 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.229158 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.229130 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.454784 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.454715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.700338 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.700300 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.834583 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.834538 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:41.931423 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:41.931400 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:42.071503 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:42.071476 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:48.392816 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:48.392785 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:48.516394 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:48.516363 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:48.667903 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:48.667818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:48.875940 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:48.875907 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:55.078757 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:55.078718 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:57:55.179108 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:57:55.179077 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:02.060995 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:02.060940 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:02.505693 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:02.505613 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:02.978766 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:02.978730 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.031898 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.031864 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.129902 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.129871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.203595 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.203562 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:58:05.203983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.203939 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" containerName="maas-api" Jul 22 11:58:05.203983 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.203973 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" containerName="maas-api" Jul 22 11:58:05.204131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.203989 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:58:05.204131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.203997 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:58:05.204131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.204109 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e5352a-d3aa-408c-925b-019a9ce13c0f" containerName="maas-api" Jul 22 11:58:05.204131 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.204127 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a62f361-f615-42a5-81a5-1acf7682a46e" containerName="istio-proxy" Jul 22 11:58:05.207244 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.207225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.210045 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.210025 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-98b5a8b0-serving-cert\"" Jul 22 11:58:05.222803 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.222777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:58:05.232232 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.232213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9g2n\" (UniqueName: \"kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.232330 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.232248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.232371 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.232354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.249289 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.249263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.333906 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.333873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9g2n\" (UniqueName: \"kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.334068 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.333939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.334119 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.334098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.334257 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:58:05.334238 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-serving-cert: secret "maas-api-e2e-mt-98b5a8b0-serving-cert" not found Jul 22 11:58:05.334332 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:58:05.334323 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls podName:edb612ea-083d-4949-a132-df907618e043 nodeName:}" failed. No retries permitted until 2026-07-22 11:58:05.834301986 +0000 UTC m=+2055.995844661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls") pod "maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" (UID: "edb612ea-083d-4949-a132-df907618e043") : secret "maas-api-e2e-mt-98b5a8b0-serving-cert" not found Jul 22 11:58:05.334533 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.334513 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.346112 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.346078 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9g2n\" (UniqueName: \"kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.695581 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.695513 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:05.837244 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.837211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:05.839729 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:05.839707 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") pod \"maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:06.120485 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.120454 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:06.358546 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.358517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:06.470149 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.470126 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:58:06.472282 ip-10-0-136-40 kubenswrapper[2569]: W0722 11:58:06.472254 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb612ea_083d_4949_a132_df907618e043.slice/crio-3360e539580962581854458eec15abc9e4d0aec87c7c076672fd2f5ba9e92c3f WatchSource:0}: Error finding container 3360e539580962581854458eec15abc9e4d0aec87c7c076672fd2f5ba9e92c3f: Status 404 returned error can't find the container with id 3360e539580962581854458eec15abc9e4d0aec87c7c076672fd2f5ba9e92c3f Jul 22 11:58:06.473532 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.473505 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:58:06.509747 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.509725 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:06.787358 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:06.787298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" event={"ID":"edb612ea-083d-4949-a132-df907618e043","Type":"ContainerStarted","Data":"3360e539580962581854458eec15abc9e4d0aec87c7c076672fd2f5ba9e92c3f"} Jul 22 11:58:07.793842 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:07.793805 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" event={"ID":"edb612ea-083d-4949-a132-df907618e043","Type":"ContainerStarted","Data":"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8"} Jul 22 11:58:07.794272 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:07.793901 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:07.815831 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:07.815787 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" podStartSLOduration=2.061897707 podStartE2EDuration="2.815773796s" totalImagesPullingTime="753.876089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:58:07.813706312 +0000 UTC m=+2057.975249009" watchObservedRunningTime="2026-07-22 11:58:07.815773796 +0000 UTC m=+2057.977316491" Jul 22 11:58:09.547991 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:09.547945 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:13.804236 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:13.804150 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:58:16.934275 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:16.934239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:21.562360 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:21.562316 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:26.004168 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:26.004133 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:58:50.367083 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:50.367051 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:58:50.377924 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:50.377903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 11:58:55.922182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:58:55.922126 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:09.068320 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.068287 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:59:09.070809 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.068492 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" podUID="edb612ea-083d-4949-a132-df907618e043" containerName="maas-api" containerID="cri-o://19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8" gracePeriod=30 Jul 22 11:59:09.111299 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.111252 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:09.320246 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.320195 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:59:09.434649 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.434624 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") pod \"edb612ea-083d-4949-a132-df907618e043\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " Jul 22 11:59:09.434811 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.434682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9g2n\" (UniqueName: \"kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n\") pod \"edb612ea-083d-4949-a132-df907618e043\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " Jul 22 11:59:09.434811 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.434719 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca\") pod \"edb612ea-083d-4949-a132-df907618e043\" (UID: \"edb612ea-083d-4949-a132-df907618e043\") " Jul 22 11:59:09.435166 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.435143 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edb612ea-083d-4949-a132-df907618e043" (UID: "edb612ea-083d-4949-a132-df907618e043"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:59:09.436796 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.436768 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edb612ea-083d-4949-a132-df907618e043" (UID: "edb612ea-083d-4949-a132-df907618e043"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:59:09.436914 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.436841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n" (OuterVolumeSpecName: "kube-api-access-l9g2n") pod "edb612ea-083d-4949-a132-df907618e043" (UID: "edb612ea-083d-4949-a132-df907618e043"). InnerVolumeSpecName "kube-api-access-l9g2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:59:09.535439 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.535412 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edb612ea-083d-4949-a132-df907618e043-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:59:09.535439 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.535436 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9g2n\" (UniqueName: \"kubernetes.io/projected/edb612ea-083d-4949-a132-df907618e043-kube-api-access-l9g2n\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:59:09.535570 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:09.535446 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb612ea-083d-4949-a132-df907618e043-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 11:59:10.097008 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.096975 2569 generic.go:356] "Generic (PLEG): container finished" podID="edb612ea-083d-4949-a132-df907618e043" containerID="19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8" exitCode=0 Jul 22 11:59:10.097414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.097021 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" Jul 22 11:59:10.097414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.097069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" event={"ID":"edb612ea-083d-4949-a132-df907618e043","Type":"ContainerDied","Data":"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8"} Jul 22 11:59:10.097414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.097123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq" event={"ID":"edb612ea-083d-4949-a132-df907618e043","Type":"ContainerDied","Data":"3360e539580962581854458eec15abc9e4d0aec87c7c076672fd2f5ba9e92c3f"} Jul 22 11:59:10.097414 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.097145 2569 scope.go:117] "RemoveContainer" containerID="19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8" Jul 22 11:59:10.107182 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.107165 2569 scope.go:117] "RemoveContainer" containerID="19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8" Jul 22 11:59:10.107484 ip-10-0-136-40 kubenswrapper[2569]: E0722 11:59:10.107464 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8\": container with ID starting with 19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8 not found: ID does not exist" containerID="19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8" Jul 22 11:59:10.107538 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.107495 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8"} err="failed to get container status \"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8\": rpc error: code = NotFound desc = could not find container \"19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8\": container with ID starting with 19d8831b21f0f27fd80ce854c54da6b9b46144d6bdfc71efee42e4d0da1b0ad8 not found: ID does not exist" Jul 22 11:59:10.122229 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.122207 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:59:10.133926 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.133901 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98b5a8b0-698c7f7688-mcpvq"] Jul 22 11:59:10.351478 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:10.351399 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb612ea-083d-4949-a132-df907618e043" path="/var/lib/kubelet/pods/edb612ea-083d-4949-a132-df907618e043/volumes" Jul 22 11:59:13.520228 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:13.520197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:15.879277 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:15.879237 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:20.988149 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:20.988086 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:22.492445 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:22.492414 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:48.197795 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:48.197751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:48.300228 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:48.300187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:57.183172 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:57.183136 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:57.479435 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:57.479311 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:57.769707 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:57.769619 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 11:59:58.936694 ip-10-0-136-40 kubenswrapper[2569]: I0722 11:59:58.936660 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:05.724673 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:05.724637 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:05.867793 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:05.867759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:06.824916 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:06.824876 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:07.330780 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:07.330744 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:07.434130 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:07.434100 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:07.849814 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:07.849780 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:08.094473 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:08.094443 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:09.424312 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:09.424278 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:15.705410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:15.705373 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:15.805467 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:15.805439 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:16.862096 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:16.862054 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:22.955857 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:22.955822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:23.061789 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:23.061758 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:23.377576 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:23.377547 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:33.830210 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:33.830169 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:34.738012 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:34.737971 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:00:55.060688 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:00:55.060643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:02.943241 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:02.943206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:03.068420 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:03.068388 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:04.084428 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:04.084390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:04.216640 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:04.216603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:04.426741 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:04.426670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:07.013119 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.013078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:07.124682 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.124652 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:07.252647 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.252613 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:01:07.253143 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.253123 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb612ea-083d-4949-a132-df907618e043" containerName="maas-api" Jul 22 12:01:07.253213 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.253149 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb612ea-083d-4949-a132-df907618e043" containerName="maas-api" Jul 22 12:01:07.253253 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.253235 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb612ea-083d-4949-a132-df907618e043" containerName="maas-api" Jul 22 12:01:07.256534 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.256516 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.258638 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.258616 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4750a4-serving-cert\"" Jul 22 12:01:07.268156 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.268091 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:01:07.354088 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.354059 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:07.388700 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.388673 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.388823 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.388739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5pw\" (UniqueName: \"kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.388823 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.388796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.489470 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.489440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.489575 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.489513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5pw\" (UniqueName: \"kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.489575 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.489533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.489685 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:01:07.489586 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-serving-cert: secret "maas-api-e2e-inf-a-4750a4-serving-cert" not found Jul 22 12:01:07.489685 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:01:07.489658 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls podName:389b927f-6d09-4ff1-ad32-65eec513b495 nodeName:}" failed. No retries permitted until 2026-07-22 12:01:07.98963568 +0000 UTC m=+2238.151178356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls") pod "maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" (UID: "389b927f-6d09-4ff1-ad32-65eec513b495") : secret "maas-api-e2e-inf-a-4750a4-serving-cert" not found Jul 22 12:01:07.490237 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.490215 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.508519 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.508492 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5pw\" (UniqueName: \"kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.993122 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.993085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:07.995763 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:07.995739 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") pod \"maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:08.002533 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:08.002490 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:08.125806 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:08.125767 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:08.167124 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:08.167095 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:08.513748 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:08.513724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:01:08.515929 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:01:08.515905 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389b927f_6d09_4ff1_ad32_65eec513b495.slice/crio-c1c7321d8463be048a1c34ae5f5b51dc7d6607b6178e088441d3758df89c8445 WatchSource:0}: Error finding container c1c7321d8463be048a1c34ae5f5b51dc7d6607b6178e088441d3758df89c8445: Status 404 returned error can't find the container with id c1c7321d8463be048a1c34ae5f5b51dc7d6607b6178e088441d3758df89c8445 Jul 22 12:01:08.652941 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:08.652900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" event={"ID":"389b927f-6d09-4ff1-ad32-65eec513b495","Type":"ContainerStarted","Data":"c1c7321d8463be048a1c34ae5f5b51dc7d6607b6178e088441d3758df89c8445"} Jul 22 12:01:09.659489 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:09.659451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" event={"ID":"389b927f-6d09-4ff1-ad32-65eec513b495","Type":"ContainerStarted","Data":"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5"} Jul 22 12:01:09.659905 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:09.659655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:09.680976 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:09.680897 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" podStartSLOduration=2.153418653 podStartE2EDuration="2.680882256s" totalImagesPullingTime="527.463603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:01:09.679306004 +0000 UTC m=+2239.840848694" watchObservedRunningTime="2026-07-22 12:01:09.680882256 +0000 UTC m=+2239.842424952" Jul 22 12:01:11.561728 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:11.561642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:11.943634 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:11.943551 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:12.302164 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:12.302130 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:13.191434 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:13.191403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:15.075661 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.075620 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:15.226181 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.226148 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:15.269938 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.269914 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:01:15.273487 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.273467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.275589 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.275566 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aa23a9-serving-cert\"" Jul 22 12:01:15.288410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.288384 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:01:15.371720 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.371661 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:15.454054 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.454023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znx5\" (UniqueName: \"kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.454193 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.454071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.454246 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.454182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.554919 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.554888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.555079 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.554946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znx5\" (UniqueName: \"kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.555079 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.555012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.555584 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.555565 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.557526 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.557508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.565274 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.565256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znx5\" (UniqueName: \"kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5\") pod \"maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.584213 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.584185 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:15.585152 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.585133 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:15.673619 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.673597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:01:15.720071 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:15.720024 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:01:15.723438 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:01:15.723409 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2115b8e2_5767_4b7c_adf5_20261d659d81.slice/crio-4de5f502baa3332fbc550ace0a77486bb072098258a85dc1934d8643c89d84f5 WatchSource:0}: Error finding container 4de5f502baa3332fbc550ace0a77486bb072098258a85dc1934d8643c89d84f5: Status 404 returned error can't find the container with id 4de5f502baa3332fbc550ace0a77486bb072098258a85dc1934d8643c89d84f5 Jul 22 12:01:16.091759 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.091730 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:16.204215 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.204190 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:16.699189 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.699159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" event={"ID":"2115b8e2-5767-4b7c-adf5-20261d659d81","Type":"ContainerStarted","Data":"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8"} Jul 22 12:01:16.699337 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.699206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" event={"ID":"2115b8e2-5767-4b7c-adf5-20261d659d81","Type":"ContainerStarted","Data":"4de5f502baa3332fbc550ace0a77486bb072098258a85dc1934d8643c89d84f5"} Jul 22 12:01:16.699337 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.699228 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:16.720938 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:16.720890 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" podStartSLOduration=1.205766386 podStartE2EDuration="1.720873444s" totalImagesPullingTime="515.107058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:01:16.719534457 +0000 UTC m=+2246.881077153" watchObservedRunningTime="2026-07-22 12:01:16.720873444 +0000 UTC m=+2246.882416143" Jul 22 12:01:19.647614 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:19.647576 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:22.708386 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:22.708356 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:01:29.933615 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:29.933573 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:30.026355 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:30.026319 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:35.740706 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:35.740669 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:37.619698 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:37.619665 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:37.838547 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:37.838514 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:41.388274 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:41.388244 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:01:56.924770 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:01:56.924731 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:00.408981 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:00.408926 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:39.689149 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:39.689104 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:43.155244 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:43.155159 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:51.411687 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.411655 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:02:51.412162 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.411917 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" podUID="389b927f-6d09-4ff1-ad32-65eec513b495" containerName="maas-api" containerID="cri-o://86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5" gracePeriod=30 Jul 22 12:02:51.457308 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.457276 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:51.665305 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.665244 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:02:51.742102 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.742069 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca\") pod \"389b927f-6d09-4ff1-ad32-65eec513b495\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " Jul 22 12:02:51.742276 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.742138 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") pod \"389b927f-6d09-4ff1-ad32-65eec513b495\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " Jul 22 12:02:51.742276 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.742167 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp5pw\" (UniqueName: \"kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw\") pod \"389b927f-6d09-4ff1-ad32-65eec513b495\" (UID: \"389b927f-6d09-4ff1-ad32-65eec513b495\") " Jul 22 12:02:51.742456 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.742429 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "389b927f-6d09-4ff1-ad32-65eec513b495" (UID: "389b927f-6d09-4ff1-ad32-65eec513b495"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:02:51.742520 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.742498 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b927f-6d09-4ff1-ad32-65eec513b495-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:02:51.744409 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.744383 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "389b927f-6d09-4ff1-ad32-65eec513b495" (UID: "389b927f-6d09-4ff1-ad32-65eec513b495"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:51.744492 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.744407 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw" (OuterVolumeSpecName: "kube-api-access-dp5pw") pod "389b927f-6d09-4ff1-ad32-65eec513b495" (UID: "389b927f-6d09-4ff1-ad32-65eec513b495"). InnerVolumeSpecName "kube-api-access-dp5pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:02:51.843620 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.843575 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389b927f-6d09-4ff1-ad32-65eec513b495-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:02:51.843620 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:51.843615 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5pw\" (UniqueName: \"kubernetes.io/projected/389b927f-6d09-4ff1-ad32-65eec513b495-kube-api-access-dp5pw\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:02:52.150716 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.150680 2569 generic.go:356] "Generic (PLEG): container finished" podID="389b927f-6d09-4ff1-ad32-65eec513b495" containerID="86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5" exitCode=0 Jul 22 12:02:52.150907 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.150727 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" Jul 22 12:02:52.150907 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.150766 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" event={"ID":"389b927f-6d09-4ff1-ad32-65eec513b495","Type":"ContainerDied","Data":"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5"} Jul 22 12:02:52.150907 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.150819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9" event={"ID":"389b927f-6d09-4ff1-ad32-65eec513b495","Type":"ContainerDied","Data":"c1c7321d8463be048a1c34ae5f5b51dc7d6607b6178e088441d3758df89c8445"} Jul 22 12:02:52.150907 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.150841 2569 scope.go:117] "RemoveContainer" containerID="86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5" Jul 22 12:02:52.159759 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.159744 2569 scope.go:117] "RemoveContainer" containerID="86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5" Jul 22 12:02:52.160036 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:02:52.160017 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5\": container with ID starting with 86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5 not found: ID does not exist" containerID="86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5" Jul 22 12:02:52.160087 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.160046 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5"} err="failed to get container status \"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5\": rpc error: code = NotFound desc = could not find container \"86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5\": container with ID starting with 86380668781b8b7a26605097109d561be56c68f692f52334be81b7eb7969caf5 not found: ID does not exist" Jul 22 12:02:52.173498 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.173475 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:02:52.178682 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.178662 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4750a4-5bd6db647b-xgfs9"] Jul 22 12:02:52.349596 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:52.349558 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389b927f-6d09-4ff1-ad32-65eec513b495" path="/var/lib/kubelet/pods/389b927f-6d09-4ff1-ad32-65eec513b495/volumes" Jul 22 12:02:55.869537 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:55.869502 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:02:56.288209 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:02:56.288127 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:03.239495 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:03.239458 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:03.338000 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:03.337944 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:03.538396 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:03.538315 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:03.808395 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:03.808369 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:04.629333 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:04.629305 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:04.729394 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:04.729362 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:14.668436 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.668399 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:03:14.668910 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.668601 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" podUID="2115b8e2-5767-4b7c-adf5-20261d659d81" containerName="maas-api" containerID="cri-o://efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8" gracePeriod=30 Jul 22 12:03:14.914193 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.914170 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:03:14.941013 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.940928 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4znx5\" (UniqueName: \"kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5\") pod \"2115b8e2-5767-4b7c-adf5-20261d659d81\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " Jul 22 12:03:14.941013 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.941006 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca\") pod \"2115b8e2-5767-4b7c-adf5-20261d659d81\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " Jul 22 12:03:14.941202 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.941084 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls\") pod \"2115b8e2-5767-4b7c-adf5-20261d659d81\" (UID: \"2115b8e2-5767-4b7c-adf5-20261d659d81\") " Jul 22 12:03:14.941426 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.941393 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2115b8e2-5767-4b7c-adf5-20261d659d81" (UID: "2115b8e2-5767-4b7c-adf5-20261d659d81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:03:14.943276 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.943250 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2115b8e2-5767-4b7c-adf5-20261d659d81" (UID: "2115b8e2-5767-4b7c-adf5-20261d659d81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:03:14.943503 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:14.943482 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5" (OuterVolumeSpecName: "kube-api-access-4znx5") pod "2115b8e2-5767-4b7c-adf5-20261d659d81" (UID: "2115b8e2-5767-4b7c-adf5-20261d659d81"). InnerVolumeSpecName "kube-api-access-4znx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:03:15.042141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.042114 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115b8e2-5767-4b7c-adf5-20261d659d81-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:03:15.042141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.042139 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2115b8e2-5767-4b7c-adf5-20261d659d81-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:03:15.042278 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.042148 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znx5\" (UniqueName: \"kubernetes.io/projected/2115b8e2-5767-4b7c-adf5-20261d659d81-kube-api-access-4znx5\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:03:15.135066 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.135043 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:15.268158 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.268106 2569 generic.go:356] "Generic (PLEG): container finished" podID="2115b8e2-5767-4b7c-adf5-20261d659d81" containerID="efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8" exitCode=0 Jul 22 12:03:15.268158 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.268150 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" Jul 22 12:03:15.268285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.268202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" event={"ID":"2115b8e2-5767-4b7c-adf5-20261d659d81","Type":"ContainerDied","Data":"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8"} Jul 22 12:03:15.268285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.268256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f" event={"ID":"2115b8e2-5767-4b7c-adf5-20261d659d81","Type":"ContainerDied","Data":"4de5f502baa3332fbc550ace0a77486bb072098258a85dc1934d8643c89d84f5"} Jul 22 12:03:15.268285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.268284 2569 scope.go:117] "RemoveContainer" containerID="efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8" Jul 22 12:03:15.277265 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.277247 2569 scope.go:117] "RemoveContainer" containerID="efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8" Jul 22 12:03:15.277567 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:03:15.277548 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8\": container with ID starting with efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8 not found: ID does not exist" containerID="efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8" Jul 22 12:03:15.277635 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.277575 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8"} err="failed to get container status \"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8\": rpc error: code = NotFound desc = could not find container \"efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8\": container with ID starting with efdb791939ffa4e6d096e461a9b8d7af3259b63820dec5d2ffb0fc2d322328a8 not found: ID does not exist" Jul 22 12:03:15.290749 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.290730 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:03:15.296332 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:15.296305 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa23a9-749cc7488f-sdd2f"] Jul 22 12:03:16.349605 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:16.349573 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2115b8e2-5767-4b7c-adf5-20261d659d81" path="/var/lib/kubelet/pods/2115b8e2-5767-4b7c-adf5-20261d659d81/volumes" Jul 22 12:03:19.019718 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:19.019681 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:19.469664 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:19.469634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:25.612297 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:25.612259 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:25.713424 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:25.713386 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:25.903643 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:25.903536 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:26.140440 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:26.140401 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:26.574652 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:26.574621 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:27.004497 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004425 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:03:27.004825 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004754 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2115b8e2-5767-4b7c-adf5-20261d659d81" containerName="maas-api" Jul 22 12:03:27.004825 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004765 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2115b8e2-5767-4b7c-adf5-20261d659d81" containerName="maas-api" Jul 22 12:03:27.004825 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004789 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389b927f-6d09-4ff1-ad32-65eec513b495" containerName="maas-api" Jul 22 12:03:27.004825 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004797 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="389b927f-6d09-4ff1-ad32-65eec513b495" containerName="maas-api" Jul 22 12:03:27.005006 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004863 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="389b927f-6d09-4ff1-ad32-65eec513b495" containerName="maas-api" Jul 22 12:03:27.005006 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.004871 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2115b8e2-5767-4b7c-adf5-20261d659d81" containerName="maas-api" Jul 22 12:03:27.007809 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.007793 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.009759 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.009740 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2dbb42-openshift-default-dockercfg-wwmnp\"" Jul 22 12:03:27.024715 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.024691 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:03:27.122567 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.122529 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:27.144337 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144468 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7mm\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144468 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144545 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144545 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144619 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144619 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144721 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.144721 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.144636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245260 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245383 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245383 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245383 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245383 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245599 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245599 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7mm\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245599 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245599 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245808 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245762 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.245808 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.245773 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.246073 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.246049 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.246168 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.246100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.246279 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.246258 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.247822 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.247806 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.247903 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.247888 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.259750 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.259677 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.260098 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.260065 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7mm\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm\") pod \"e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.262444 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.262347 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:27.317483 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.317460 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:27.462375 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.462349 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:03:27.464555 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:03:27.464527 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee99c243_be29_4e16_998d_347c1ac34789.slice/crio-adde65d1b62285978e3b41f553bbe495d943b951cf8bb5cf72e0f3b91f482bf9 WatchSource:0}: Error finding container adde65d1b62285978e3b41f553bbe495d943b951cf8bb5cf72e0f3b91f482bf9: Status 404 returned error can't find the container with id adde65d1b62285978e3b41f553bbe495d943b951cf8bb5cf72e0f3b91f482bf9 Jul 22 12:03:27.466606 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.466577 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:27.466671 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.466642 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:27.466709 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:27.466669 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:28.333570 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:28.333537 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerStarted","Data":"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8"} Jul 22 12:03:28.333926 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:28.333584 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerStarted","Data":"adde65d1b62285978e3b41f553bbe495d943b951cf8bb5cf72e0f3b91f482bf9"} Jul 22 12:03:28.359190 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:28.359145 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podStartSLOduration=2.359130028 podStartE2EDuration="2.359130028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:03:28.356750235 +0000 UTC m=+2378.518292932" watchObservedRunningTime="2026-07-22 12:03:28.359130028 +0000 UTC m=+2378.520672724" Jul 22 12:03:29.317649 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:29.317615 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:29.322487 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:29.322464 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:29.338787 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:29.338754 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:29.339583 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:29.339546 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:30.454043 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.454011 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:03:30.457299 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.457285 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.459259 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.459234 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2dbb42-serving-cert\"" Jul 22 12:03:30.469426 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.469404 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:03:30.576736 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.576696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.576926 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.576743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtng\" (UniqueName: \"kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.576926 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.576826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.628093 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.628062 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:30.678221 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.678194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.678329 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.678228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtng\" (UniqueName: \"kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.678329 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.678262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.678445 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:03:30.678341 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-serving-cert: secret "maas-api-e2e-api-a-2dbb42-serving-cert" not found Jul 22 12:03:30.678445 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:03:30.678407 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls podName:1bd81135-c11c-4847-bd1b-b4d6cc2a78e4 nodeName:}" failed. No retries permitted until 2026-07-22 12:03:31.178389179 +0000 UTC m=+2381.339931854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls") pod "maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" (UID: "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4") : secret "maas-api-e2e-api-a-2dbb42-serving-cert" not found Jul 22 12:03:30.678842 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.678824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.687215 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.687192 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtng\" (UniqueName: \"kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:30.729940 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.729878 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:30.827329 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:30.827298 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:31.100545 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.100521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:31.183451 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.183428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:31.185872 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.185846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") pod \"maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:31.199851 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.199827 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:31.368243 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.368161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:31.512196 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.512168 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:03:31.514606 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:03:31.514581 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd81135_c11c_4847_bd1b_b4d6cc2a78e4.slice/crio-f4a508147e3359a17abbae088a76df2244370fbdd0bde70c950b2245b7951238 WatchSource:0}: Error finding container f4a508147e3359a17abbae088a76df2244370fbdd0bde70c950b2245b7951238: Status 404 returned error can't find the container with id f4a508147e3359a17abbae088a76df2244370fbdd0bde70c950b2245b7951238 Jul 22 12:03:31.516015 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:31.515996 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:03:32.353092 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:32.353055 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" event={"ID":"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4","Type":"ContainerStarted","Data":"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2"} Jul 22 12:03:32.353289 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:32.353119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" event={"ID":"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4","Type":"ContainerStarted","Data":"f4a508147e3359a17abbae088a76df2244370fbdd0bde70c950b2245b7951238"} Jul 22 12:03:32.353289 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:32.353181 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:32.371307 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:32.371255 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" podStartSLOduration=1.8541569629999999 podStartE2EDuration="2.371240413s" totalImagesPullingTime="517.08345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:03:32.368356792 +0000 UTC m=+2382.529899487" watchObservedRunningTime="2026-07-22 12:03:32.371240413 +0000 UTC m=+2382.532783166" Jul 22 12:03:34.988674 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:34.988643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:35.584374 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:35.584333 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:36.002206 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:36.002119 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:36.231494 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:36.231463 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:38.362999 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:38.362972 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:03:39.574873 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.574840 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:39.672638 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.672607 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:03:39.681296 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.681267 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:39.690977 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.688500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.696192 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.691647 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:03:39.696192 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.693880 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f39058-serving-cert\"" Jul 22 12:03:39.758882 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.758847 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:39.763209 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.763188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.763304 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.763240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.763360 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.763340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xsw\" (UniqueName: \"kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.864439 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.864353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xsw\" (UniqueName: \"kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.864439 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.864422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.864644 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.864467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.864644 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:03:39.864569 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-serving-cert: secret "maas-api-e2e-api-b-f39058-serving-cert" not found Jul 22 12:03:39.864644 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:03:39.864638 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls podName:6d24d621-f64b-47f3-a035-7d99ea324966 nodeName:}" failed. No retries permitted until 2026-07-22 12:03:40.364621461 +0000 UTC m=+2390.526164135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls") pod "maas-api-e2e-api-b-f39058-846c45b596-hstxt" (UID: "6d24d621-f64b-47f3-a035-7d99ea324966") : secret "maas-api-e2e-api-b-f39058-serving-cert" not found Jul 22 12:03:39.865090 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.865071 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:39.891644 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:39.891609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xsw\" (UniqueName: \"kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:40.250822 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.250721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:40.368897 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.368866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:40.371507 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.371478 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") pod \"maas-api-e2e-api-b-f39058-846c45b596-hstxt\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:40.611159 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.611123 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:40.801048 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.786611 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:03:40.801048 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:03:40.800610 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d24d621_f64b_47f3_a035_7d99ea324966.slice/crio-62cc74155dc01fec2e0835ef5392a3e8ddea825df4e7e863fe4d0f290d46be3e WatchSource:0}: Error finding container 62cc74155dc01fec2e0835ef5392a3e8ddea825df4e7e863fe4d0f290d46be3e: Status 404 returned error can't find the container with id 62cc74155dc01fec2e0835ef5392a3e8ddea825df4e7e863fe4d0f290d46be3e Jul 22 12:03:40.848415 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.848382 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:40.962452 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:40.962355 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:41.407757 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:41.407723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" event={"ID":"6d24d621-f64b-47f3-a035-7d99ea324966","Type":"ContainerStarted","Data":"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e"} Jul 22 12:03:41.407986 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:41.407773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" event={"ID":"6d24d621-f64b-47f3-a035-7d99ea324966","Type":"ContainerStarted","Data":"62cc74155dc01fec2e0835ef5392a3e8ddea825df4e7e863fe4d0f290d46be3e"} Jul 22 12:03:41.407986 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:41.407866 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:44.183325 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.183296 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:44.339895 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.339861 2569 patch_prober.go:28] interesting pod/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 12:03:44.340071 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.339911 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 12:03:44.424829 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.424795 2569 generic.go:356] "Generic (PLEG): container finished" podID="ee99c243-be29-4e16-998d-347c1ac34789" containerID="6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8" exitCode=0 Jul 22 12:03:44.425014 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.424871 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerDied","Data":"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8"} Jul 22 12:03:44.425262 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.425247 2569 scope.go:117] "RemoveContainer" containerID="6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8" Jul 22 12:03:44.426269 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.426245 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:44.426337 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.426312 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:44.426374 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.426344 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:03:44.448802 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:44.448759 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" podStartSLOduration=5.052682508 podStartE2EDuration="5.448743801s" totalImagesPullingTime="396.061293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:03:41.425185074 +0000 UTC m=+2391.586727817" watchObservedRunningTime="2026-07-22 12:03:44.448743801 +0000 UTC m=+2394.610286561" Jul 22 12:03:45.431633 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:45.431600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerStarted","Data":"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d"} Jul 22 12:03:46.318133 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:46.318098 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:46.318384 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:46.318287 2569 patch_prober.go:28] interesting pod/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 12:03:46.318384 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:46.318332 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 12:03:47.318051 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:47.318018 2569 patch_prober.go:28] interesting pod/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 12:03:47.318437 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:47.318075 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 12:03:47.417702 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:47.417668 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:03:48.318514 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:48.318477 2569 patch_prober.go:28] interesting pod/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 12:03:48.318900 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:48.318532 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 12:03:49.322377 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:49.322348 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:49.322827 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:49.322619 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:49.323346 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:49.323329 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:03:50.396479 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:50.396451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 12:03:50.406310 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:50.406290 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 12:03:52.063347 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:52.063316 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:03:52.182215 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:03:52.182179 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:01.614489 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:01.614457 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:01.934435 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:01.934353 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:07.579543 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.579510 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:04:07.579944 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.579693 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" podUID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" containerName="maas-api" containerID="cri-o://f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2" gracePeriod=30 Jul 22 12:04:07.633501 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.633467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:07.832591 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.832529 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:04:07.923608 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.923583 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca\") pod \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " Jul 22 12:04:07.923734 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.923639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjtng\" (UniqueName: \"kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng\") pod \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " Jul 22 12:04:07.923734 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.923677 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") pod \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\" (UID: \"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4\") " Jul 22 12:04:07.924006 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.923971 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" (UID: "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:04:07.925900 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.925877 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" (UID: "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:04:07.926004 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:07.925902 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng" (OuterVolumeSpecName: "kube-api-access-fjtng") pod "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" (UID: "1bd81135-c11c-4847-bd1b-b4d6cc2a78e4"). InnerVolumeSpecName "kube-api-access-fjtng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:04:08.025149 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.025126 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:08.025149 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.025147 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtng\" (UniqueName: \"kubernetes.io/projected/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-kube-api-access-fjtng\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:08.025270 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.025157 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:08.541028 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.540998 2569 generic.go:356] "Generic (PLEG): container finished" podID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" containerID="f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2" exitCode=0 Jul 22 12:04:08.541143 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.541048 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" Jul 22 12:04:08.541143 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.541054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" event={"ID":"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4","Type":"ContainerDied","Data":"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2"} Jul 22 12:04:08.541143 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.541095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk" event={"ID":"1bd81135-c11c-4847-bd1b-b4d6cc2a78e4","Type":"ContainerDied","Data":"f4a508147e3359a17abbae088a76df2244370fbdd0bde70c950b2245b7951238"} Jul 22 12:04:08.541143 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.541119 2569 scope.go:117] "RemoveContainer" containerID="f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2" Jul 22 12:04:08.550311 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.550286 2569 scope.go:117] "RemoveContainer" containerID="f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2" Jul 22 12:04:08.550588 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:04:08.550564 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2\": container with ID starting with f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2 not found: ID does not exist" containerID="f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2" Jul 22 12:04:08.550683 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.550594 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2"} err="failed to get container status \"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2\": rpc error: code = NotFound desc = could not find container \"f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2\": container with ID starting with f27eedba2ae23fd2ee1ca89c0570ee2020e22ecf7e6f6e26507b444fd64ecbf2 not found: ID does not exist" Jul 22 12:04:08.567806 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.567781 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:04:08.574800 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:08.574776 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2dbb42-69b5f98f6-kc6wk"] Jul 22 12:04:10.350066 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:10.350029 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" path="/var/lib/kubelet/pods/1bd81135-c11c-4847-bd1b-b4d6cc2a78e4/volumes" Jul 22 12:04:12.351745 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:12.351707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:12.782009 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:12.781880 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:19.098434 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.098401 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:19.202857 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.202822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:19.362724 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.362650 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:04:19.362903 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.362877 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" containerID="cri-o://e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d" gracePeriod=30 Jul 22 12:04:19.390227 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.390198 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:19.630228 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:19.630139 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:24.616833 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.616813 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:04:24.619196 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.619169 2569 generic.go:356] "Generic (PLEG): container finished" podID="ee99c243-be29-4e16-998d-347c1ac34789" containerID="e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d" exitCode=0 Jul 22 12:04:24.619319 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.619203 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" Jul 22 12:04:24.619319 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.619249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerDied","Data":"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d"} Jul 22 12:04:24.619395 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.619323 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf" event={"ID":"ee99c243-be29-4e16-998d-347c1ac34789","Type":"ContainerDied","Data":"adde65d1b62285978e3b41f553bbe495d943b951cf8bb5cf72e0f3b91f482bf9"} Jul 22 12:04:24.619395 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.619346 2569 scope.go:117] "RemoveContainer" containerID="e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d" Jul 22 12:04:24.627475 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.627457 2569 scope.go:117] "RemoveContainer" containerID="6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8" Jul 22 12:04:24.635224 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.635208 2569 scope.go:117] "RemoveContainer" containerID="e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d" Jul 22 12:04:24.635465 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:04:24.635445 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d\": container with ID starting with e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d not found: ID does not exist" containerID="e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d" Jul 22 12:04:24.635551 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.635472 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d"} err="failed to get container status \"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d\": rpc error: code = NotFound desc = could not find container \"e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d\": container with ID starting with e401337089982617f4ec30cc27fc2d2d36cfa6af9109b7cd5ac9662063cbc61d not found: ID does not exist" Jul 22 12:04:24.635551 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.635488 2569 scope.go:117] "RemoveContainer" containerID="6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8" Jul 22 12:04:24.635703 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:04:24.635687 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8\": container with ID starting with 6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8 not found: ID does not exist" containerID="6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8" Jul 22 12:04:24.635745 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.635708 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8"} err="failed to get container status \"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8\": rpc error: code = NotFound desc = could not find container \"6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8\": container with ID starting with 6563aad45d675b6d0e5367d65fcb2a4b4092d9189462781f07433381227b13d8 not found: ID does not exist" Jul 22 12:04:24.765463 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765432 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765631 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765485 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz7mm\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765631 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765506 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765631 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765542 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765631 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765580 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765631 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765624 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765882 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765649 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765882 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765882 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765739 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data\") pod \"ee99c243-be29-4e16-998d-347c1ac34789\" (UID: \"ee99c243-be29-4e16-998d-347c1ac34789\") " Jul 22 12:04:24.765882 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765766 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:04:24.766181 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.765917 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:04:24.766181 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.766161 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.766285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.766182 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-credential-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.766285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.766215 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:04:24.766444 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.766399 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:04:24.767067 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.767044 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:04:24.767904 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.767873 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm" (OuterVolumeSpecName: "kube-api-access-hz7mm") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "kube-api-access-hz7mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:04:24.768543 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.768520 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:04:24.768733 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.768713 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 12:04:24.768733 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.768716 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee99c243-be29-4e16-998d-347c1ac34789" (UID: "ee99c243-be29-4e16-998d-347c1ac34789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:04:24.867497 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867454 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee99c243-be29-4e16-998d-347c1ac34789-istio-podinfo\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867497 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867488 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-workload-certs\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867497 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867497 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-istio-token\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867497 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867505 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-envoy\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867787 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867513 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee99c243-be29-4e16-998d-347c1ac34789-istio-data\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867787 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867521 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7mm\" (UniqueName: \"kubernetes.io/projected/ee99c243-be29-4e16-998d-347c1ac34789-kube-api-access-hz7mm\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.867787 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.867531 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee99c243-be29-4e16-998d-347c1ac34789-istiod-ca-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:24.943091 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.943057 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:04:24.948122 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:24.948099 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2dbb42-openshift-default-dfbfd479-v6bwf"] Jul 22 12:04:26.349933 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:26.349897 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee99c243-be29-4e16-998d-347c1ac34789" path="/var/lib/kubelet/pods/ee99c243-be29-4e16-998d-347c1ac34789/volumes" Jul 22 12:04:35.302690 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.302656 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:04:35.303332 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.302881 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" podUID="6d24d621-f64b-47f3-a035-7d99ea324966" containerName="maas-api" containerID="cri-o://b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e" gracePeriod=30 Jul 22 12:04:35.360032 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.359999 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:35.555450 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.555395 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:04:35.668265 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.668240 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") pod \"6d24d621-f64b-47f3-a035-7d99ea324966\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " Jul 22 12:04:35.668420 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.668311 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca\") pod \"6d24d621-f64b-47f3-a035-7d99ea324966\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " Jul 22 12:04:35.668420 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.668355 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xsw\" (UniqueName: \"kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw\") pod \"6d24d621-f64b-47f3-a035-7d99ea324966\" (UID: \"6d24d621-f64b-47f3-a035-7d99ea324966\") " Jul 22 12:04:35.668633 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.668611 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d24d621-f64b-47f3-a035-7d99ea324966" (UID: "6d24d621-f64b-47f3-a035-7d99ea324966"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:04:35.670529 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.670507 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw" (OuterVolumeSpecName: "kube-api-access-t7xsw") pod "6d24d621-f64b-47f3-a035-7d99ea324966" (UID: "6d24d621-f64b-47f3-a035-7d99ea324966"). InnerVolumeSpecName "kube-api-access-t7xsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:04:35.670626 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.670593 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d24d621-f64b-47f3-a035-7d99ea324966" (UID: "6d24d621-f64b-47f3-a035-7d99ea324966"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:04:35.674057 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.674032 2569 generic.go:356] "Generic (PLEG): container finished" podID="6d24d621-f64b-47f3-a035-7d99ea324966" containerID="b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e" exitCode=0 Jul 22 12:04:35.674141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.674069 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" Jul 22 12:04:35.674141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.674082 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" event={"ID":"6d24d621-f64b-47f3-a035-7d99ea324966","Type":"ContainerDied","Data":"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e"} Jul 22 12:04:35.674141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.674134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt" event={"ID":"6d24d621-f64b-47f3-a035-7d99ea324966","Type":"ContainerDied","Data":"62cc74155dc01fec2e0835ef5392a3e8ddea825df4e7e863fe4d0f290d46be3e"} Jul 22 12:04:35.674246 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.674163 2569 scope.go:117] "RemoveContainer" containerID="b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e" Jul 22 12:04:35.686996 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.686977 2569 scope.go:117] "RemoveContainer" containerID="b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e" Jul 22 12:04:35.687227 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:04:35.687210 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e\": container with ID starting with b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e not found: ID does not exist" containerID="b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e" Jul 22 12:04:35.687289 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.687235 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e"} err="failed to get container status \"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e\": rpc error: code = NotFound desc = could not find container \"b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e\": container with ID starting with b608adfd836a1bd0a4d618ff3a844ea0e1feca36f5bd3adcf907360e9109e99e not found: ID does not exist" Jul 22 12:04:35.700392 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.700370 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:04:35.707828 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.707809 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f39058-846c45b596-hstxt"] Jul 22 12:04:35.769390 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.769363 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d24d621-f64b-47f3-a035-7d99ea324966-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:35.769390 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.769387 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xsw\" (UniqueName: \"kubernetes.io/projected/6d24d621-f64b-47f3-a035-7d99ea324966-kube-api-access-t7xsw\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:35.769520 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:35.769402 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d24d621-f64b-47f3-a035-7d99ea324966-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:04:36.349873 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:36.349841 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d24d621-f64b-47f3-a035-7d99ea324966" path="/var/lib/kubelet/pods/6d24d621-f64b-47f3-a035-7d99ea324966/volumes" Jul 22 12:04:37.674473 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:37.674444 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:39.852427 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:39.852389 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:40.241136 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:40.241047 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:46.284295 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:46.284260 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:46.381333 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:46.381302 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:46.597505 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:46.597474 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:46.829036 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:46.829004 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:47.186681 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.186642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:47.490068 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490001 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490333 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d24d621-f64b-47f3-a035-7d99ea324966" containerName="maas-api" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490344 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d24d621-f64b-47f3-a035-7d99ea324966" containerName="maas-api" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490368 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490373 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490380 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" containerName="maas-api" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490386 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" containerName="maas-api" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490392 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490397 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490680 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490448 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d24d621-f64b-47f3-a035-7d99ea324966" containerName="maas-api" Jul 22 12:04:47.490680 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490457 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490680 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490462 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee99c243-be29-4e16-998d-347c1ac34789" containerName="istio-proxy" Jul 22 12:04:47.490680 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.490472 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd81135-c11c-4847-bd1b-b4d6cc2a78e4" containerName="maas-api" Jul 22 12:04:47.494740 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.494721 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.497195 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.497176 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fc3650-openshift-default-dockercfg-bsrd6\"" Jul 22 12:04:47.506221 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.506201 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:04:47.561760 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.561868 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.561868 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561799 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.561948 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcmbq\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.561948 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.561948 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.562104 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.561978 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.562104 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.562030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.562104 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.562049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662504 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662693 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662693 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662693 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcmbq\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662861 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.662861 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.662773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663103 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663103 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663253 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663253 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663253 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663138 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663538 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663513 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663619 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.663619 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.663608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.665586 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.665556 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.665740 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.665723 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.668532 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.668497 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:47.673312 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.673288 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.673916 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.673895 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcmbq\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq\") pod \"e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.804640 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.804621 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:47.939582 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.939543 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:04:47.941580 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:04:47.941556 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75ef404_49f7_4534_8224_d11ce8dd3d37.slice/crio-a21dcbd93534b2c53cd623fc25ecb942eaa75c2acd7b9c3d681b4f5a73880066 WatchSource:0}: Error finding container a21dcbd93534b2c53cd623fc25ecb942eaa75c2acd7b9c3d681b4f5a73880066: Status 404 returned error can't find the container with id a21dcbd93534b2c53cd623fc25ecb942eaa75c2acd7b9c3d681b4f5a73880066 Jul 22 12:04:47.944015 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.943988 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:04:47.944102 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.944051 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:04:47.944102 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:47.944079 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:04:48.282343 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:48.282266 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:48.739788 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:48.739750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerStarted","Data":"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f"} Jul 22 12:04:48.740208 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:48.739815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerStarted","Data":"a21dcbd93534b2c53cd623fc25ecb942eaa75c2acd7b9c3d681b4f5a73880066"} Jul 22 12:04:48.760742 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:48.760696 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podStartSLOduration=1.7606802049999999 podStartE2EDuration="1.760680205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:04:48.759697866 +0000 UTC m=+2458.921240563" watchObservedRunningTime="2026-07-22 12:04:48.760680205 +0000 UTC m=+2458.922222902" Jul 22 12:04:48.805016 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:48.804982 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:49.218529 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:49.218500 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:49.334410 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:49.334378 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:49.809687 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:49.809660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:50.686026 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.685989 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:50.755223 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.755198 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:50.756040 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.756021 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:04:50.802283 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.802248 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:04:50.807228 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.807204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.809275 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.809256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fc3650-serving-cert\"" Jul 22 12:04:50.817034 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.817012 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:04:50.817748 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.817727 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:50.891686 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.891657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.891805 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.891704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfr4s\" (UniqueName: \"kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.891805 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.891721 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.921730 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.921695 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:50.992367 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.992339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.992507 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.992383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfr4s\" (UniqueName: \"kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.992507 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.992403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.993062 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.993038 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:50.994989 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:50.994969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:51.001429 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.001410 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfr4s\" (UniqueName: \"kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s\") pod \"maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:51.019282 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.019263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:51.119601 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.119548 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:51.259284 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.259260 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:04:51.261180 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:04:51.261153 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14f3297_3b62_4a63_b2ae_53ed7af4289b.slice/crio-782aec0d3e0f5fb9e5f25204a94f8fc48b2ee388f1fcfcafa41a876559833807 WatchSource:0}: Error finding container 782aec0d3e0f5fb9e5f25204a94f8fc48b2ee388f1fcfcafa41a876559833807: Status 404 returned error can't find the container with id 782aec0d3e0f5fb9e5f25204a94f8fc48b2ee388f1fcfcafa41a876559833807 Jul 22 12:04:51.529476 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.529369 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:51.641866 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.641831 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:51.761561 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:51.761525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" event={"ID":"d14f3297-3b62-4a63-b2ae-53ed7af4289b","Type":"ContainerStarted","Data":"782aec0d3e0f5fb9e5f25204a94f8fc48b2ee388f1fcfcafa41a876559833807"} Jul 22 12:04:52.768354 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:52.768274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" event={"ID":"d14f3297-3b62-4a63-b2ae-53ed7af4289b","Type":"ContainerStarted","Data":"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097"} Jul 22 12:04:52.789433 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:52.789383 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" podStartSLOduration=1.751437445 podStartE2EDuration="2.789363237s" totalImagesPullingTime="1.037925792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:04:52.788264758 +0000 UTC m=+2462.949807453" watchObservedRunningTime="2026-07-22 12:04:52.789363237 +0000 UTC m=+2462.950905934" Jul 22 12:04:53.773324 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:53.773293 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:55.584398 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:55.584361 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:55.687541 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:55.687507 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:55.851920 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:55.851825 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:56.162702 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:56.162610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:59.201139 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:59.201096 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:59.294785 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:59.294752 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:59.368227 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:59.368194 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:04:59.787606 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:59.787574 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:04:59.910024 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:04:59.909986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:00.037941 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:00.037865 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:00.132298 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:00.132273 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:03.695411 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:03.695366 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:06.797134 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:06.797087 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 12:05:06.797679 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:06.797147 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 12:05:07.845270 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.845234 2569 generic.go:356] "Generic (PLEG): container finished" podID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerID="3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f" exitCode=0 Jul 22 12:05:07.845669 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.845312 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerDied","Data":"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f"} Jul 22 12:05:07.845716 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.845684 2569 scope.go:117] "RemoveContainer" containerID="3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f" Jul 22 12:05:07.846644 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.846620 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:05:07.846698 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.846686 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:05:07.846732 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:07.846719 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 12:05:08.852879 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:08.852836 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerStarted","Data":"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c"} Jul 22 12:05:09.805020 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:09.804945 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:05:09.805242 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:09.805217 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 12:05:09.805293 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:09.805276 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 12:05:10.805725 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:10.805681 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 12:05:10.806163 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:10.805740 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 12:05:11.805662 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:11.805621 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 12:05:11.805823 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:11.805681 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 12:05:12.805734 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:12.805698 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 12:05:12.805909 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:12.805753 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 12:05:12.851290 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:12.851257 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:12.952203 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:12.952167 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:13.808904 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:13.808874 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:05:13.809296 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:13.809274 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:05:13.810001 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:13.809981 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:05:16.643965 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:16.643918 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:20.897902 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:20.897868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:20.998985 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:20.998932 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:05:43.532327 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:05:43.532248 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:06:48.950478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:06:48.950433 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:07:35.788734 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:07:35.788634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:07:36.721896 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:07:36.721867 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:07:54.959378 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:07:54.959342 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:07:56.510531 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:07:56.510497 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:08:12.673403 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:08:12.673362 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:08:15.196521 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:08:15.196485 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:08:50.425168 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:08:50.425137 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 12:08:50.434464 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:08:50.434446 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 12:09:00.513643 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:00.513613 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:00.617509 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:00.617479 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:21.625907 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:21.625871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:22.867560 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:22.867526 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:50.170117 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:50.170085 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:50.270360 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:50.270311 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:09:55.640621 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:09:55.640587 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:13.819069 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:13.818986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:13.919378 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:13.919344 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:21.981938 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:21.981905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:22.081851 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:22.081820 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:37.932390 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:37.932351 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:43.010067 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:43.010034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:10:43.896042 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:10:43.896008 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:04.216594 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:04.216555 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:04.307739 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:04.307709 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:14.079037 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.079003 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:11:14.079641 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.079246 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" podUID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" containerName="maas-api" containerID="cri-o://58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097" gracePeriod=30 Jul 22 12:11:14.100930 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.100904 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:14.335211 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.335150 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:11:14.395214 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.395185 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca\") pod \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " Jul 22 12:11:14.395351 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.395229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfr4s\" (UniqueName: \"kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s\") pod \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " Jul 22 12:11:14.395351 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.395277 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls\") pod \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\" (UID: \"d14f3297-3b62-4a63-b2ae-53ed7af4289b\") " Jul 22 12:11:14.395606 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.395577 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d14f3297-3b62-4a63-b2ae-53ed7af4289b" (UID: "d14f3297-3b62-4a63-b2ae-53ed7af4289b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:11:14.397467 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.397444 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s" (OuterVolumeSpecName: "kube-api-access-xfr4s") pod "d14f3297-3b62-4a63-b2ae-53ed7af4289b" (UID: "d14f3297-3b62-4a63-b2ae-53ed7af4289b"). InnerVolumeSpecName "kube-api-access-xfr4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:11:14.397564 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.397508 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d14f3297-3b62-4a63-b2ae-53ed7af4289b" (UID: "d14f3297-3b62-4a63-b2ae-53ed7af4289b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:11:14.496910 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.496887 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f3297-3b62-4a63-b2ae-53ed7af4289b-maas-api-tls\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:14.496910 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.496910 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f3297-3b62-4a63-b2ae-53ed7af4289b-openshift-service-ca\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:14.497079 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.496922 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfr4s\" (UniqueName: \"kubernetes.io/projected/d14f3297-3b62-4a63-b2ae-53ed7af4289b-kube-api-access-xfr4s\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:14.608478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.608401 2569 generic.go:356] "Generic (PLEG): container finished" podID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" containerID="58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097" exitCode=0 Jul 22 12:11:14.608478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.608446 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" Jul 22 12:11:14.608478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.608455 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" event={"ID":"d14f3297-3b62-4a63-b2ae-53ed7af4289b","Type":"ContainerDied","Data":"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097"} Jul 22 12:11:14.608707 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.608496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r" event={"ID":"d14f3297-3b62-4a63-b2ae-53ed7af4289b","Type":"ContainerDied","Data":"782aec0d3e0f5fb9e5f25204a94f8fc48b2ee388f1fcfcafa41a876559833807"} Jul 22 12:11:14.608707 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.608518 2569 scope.go:117] "RemoveContainer" containerID="58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097" Jul 22 12:11:14.618227 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.618209 2569 scope.go:117] "RemoveContainer" containerID="58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097" Jul 22 12:11:14.618478 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:11:14.618461 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097\": container with ID starting with 58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097 not found: ID does not exist" containerID="58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097" Jul 22 12:11:14.618556 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.618485 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097"} err="failed to get container status \"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097\": rpc error: code = NotFound desc = could not find container \"58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097\": container with ID starting with 58fc962e6411322f9b5ea59132e3c5564f98dc2eb5dbeca3ef67c5fc352fb097 not found: ID does not exist" Jul 22 12:11:14.631570 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.631545 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:11:14.633618 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:14.633598 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fc3650-6d6cc598c8-cr95r"] Jul 22 12:11:16.350157 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:16.350122 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" path="/var/lib/kubelet/pods/d14f3297-3b62-4a63-b2ae-53ed7af4289b/volumes" Jul 22 12:11:18.667444 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:18.667406 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:19.093422 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:19.093394 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:20.338625 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:20.338593 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:20.398876 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:20.398844 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:25.319409 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:25.319372 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:25.385189 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:25.385163 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:25.582927 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:25.582841 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:11:25.583170 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:25.583139 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" containerID="cri-o://7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c" gracePeriod=30 Jul 22 12:11:26.022176 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:26.022099 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:26.088791 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:26.088758 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:26.673228 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:26.673192 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:26.800375 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:26.800346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:28.812187 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:28.812153 2569 patch_prober.go:28] interesting pod/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 12:11:28.812564 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:28.812215 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 12:11:30.837642 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.837612 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:11:30.938799 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938705 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.938799 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938745 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.938799 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938794 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938821 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcmbq\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938903 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.938938 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939033 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939110 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data\") pod \"b75ef404-49f7-4534-8224-d11ce8dd3d37\" (UID: \"b75ef404-49f7-4534-8224-d11ce8dd3d37\") " Jul 22 12:11:30.939388 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939135 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:11:30.939388 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939190 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:11:30.939388 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939353 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:11:30.939527 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939435 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:11:30.939527 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939439 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:30.939527 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939464 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-workload-certs\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:30.939527 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.939478 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-credential-socket\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:30.940204 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.940177 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data" (OuterVolumeSpecName: "istio-data") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:11:30.941120 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.941100 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 12:11:30.941599 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.941573 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token" (OuterVolumeSpecName: "istio-token") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:11:30.941704 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.941605 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq" (OuterVolumeSpecName: "kube-api-access-qcmbq") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "kube-api-access-qcmbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:11:30.941704 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:30.941663 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b75ef404-49f7-4534-8224-d11ce8dd3d37" (UID: "b75ef404-49f7-4534-8224-d11ce8dd3d37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:11:31.040450 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040424 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-token\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.040450 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040445 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75ef404-49f7-4534-8224-d11ce8dd3d37-istiod-ca-cert\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.040450 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040455 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-data\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.040639 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040462 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-podinfo\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.040639 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040470 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75ef404-49f7-4534-8224-d11ce8dd3d37-istio-envoy\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.040639 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.040478 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcmbq\" (UniqueName: \"kubernetes.io/projected/b75ef404-49f7-4534-8224-d11ce8dd3d37-kube-api-access-qcmbq\") on node \"ip-10-0-136-40.ec2.internal\" DevicePath \"\"" Jul 22 12:11:31.689148 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.689111 2569 generic.go:356] "Generic (PLEG): container finished" podID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerID="7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c" exitCode=0 Jul 22 12:11:31.689334 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.689176 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" Jul 22 12:11:31.689334 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.689186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerDied","Data":"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c"} Jul 22 12:11:31.689334 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.689229 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65" event={"ID":"b75ef404-49f7-4534-8224-d11ce8dd3d37","Type":"ContainerDied","Data":"a21dcbd93534b2c53cd623fc25ecb942eaa75c2acd7b9c3d681b4f5a73880066"} Jul 22 12:11:31.689334 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.689249 2569 scope.go:117] "RemoveContainer" containerID="7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c" Jul 22 12:11:31.698193 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.698174 2569 scope.go:117] "RemoveContainer" containerID="3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f" Jul 22 12:11:31.706400 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.706251 2569 scope.go:117] "RemoveContainer" containerID="7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c" Jul 22 12:11:31.706550 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:11:31.706531 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c\": container with ID starting with 7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c not found: ID does not exist" containerID="7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c" Jul 22 12:11:31.706591 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.706559 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c"} err="failed to get container status \"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c\": rpc error: code = NotFound desc = could not find container \"7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c\": container with ID starting with 7d110d3f619facc2bcde459afcebe20aad58128741ece6310650df020ce7fd2c not found: ID does not exist" Jul 22 12:11:31.706591 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.706577 2569 scope.go:117] "RemoveContainer" containerID="3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f" Jul 22 12:11:31.706797 ip-10-0-136-40 kubenswrapper[2569]: E0722 12:11:31.706779 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f\": container with ID starting with 3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f not found: ID does not exist" containerID="3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f" Jul 22 12:11:31.706839 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.706804 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f"} err="failed to get container status \"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f\": rpc error: code = NotFound desc = could not find container \"3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f\": container with ID starting with 3f55b54a20ec76221110f49c6ab7f66b0f44eef1e3d710f06596e890df6f311f not found: ID does not exist" Jul 22 12:11:31.712653 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.712629 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:11:31.719372 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:31.719351 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fc3650-openshift-default-78b697fcf9-wfn65"] Jul 22 12:11:32.350463 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:32.350431 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" path="/var/lib/kubelet/pods/b75ef404-49f7-4534-8224-d11ce8dd3d37/volumes" Jul 22 12:11:36.576680 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:36.576652 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:41.068795 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:41.068711 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:41.486397 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:41.486315 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:47.904488 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:47.904434 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:47.977478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:47.977439 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:48.258372 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:48.258283 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:48.497047 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:48.497014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:11:55.022427 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:55.022395 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74bd8b57c-9whxb_caa60025-11c9-437d-92b0-c982b1a21f37/manager/0.log" Jul 22 12:11:55.413070 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:55.413042 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m_6e9df6e6-bcda-4214-b8d1-13fedd123f54/copy-manifests/0.log" Jul 22 12:11:55.421157 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:55.421136 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m_6e9df6e6-bcda-4214-b8d1-13fedd123f54/manager/0.log" Jul 22 12:11:56.064504 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:56.064476 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d896bf67-vws88_c80c1a97-201f-43df-b995-769796fc8bed/maas-api/0.log" Jul 22 12:11:56.417982 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:56.417889 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ncktw_79383aa6-ad3b-4a6d-bfc6-665e2b15a653/postgres/0.log" Jul 22 12:11:58.695105 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:58.695073 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t2zlp_9165cfd9-1d39-455e-8cf0-e9c858320342/manager/0.log" Jul 22 12:11:59.077347 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:59.077317 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dbmnmv_90e859e1-8863-422c-9a49-d8749e5b334f/istio-proxy/0.log" Jul 22 12:11:59.210535 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:59.210505 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gdj7_68288dc1-7b9d-49d8-82e4-20b7d246e510/discovery/0.log" Jul 22 12:11:59.324455 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:59.324432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ffd4cd5bc-fjtgs_370974f3-afe9-436b-9bca-a7fd3c84cd02/kube-auth-proxy/0.log" Jul 22 12:11:59.582068 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:59.582044 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/1.log" Jul 22 12:11:59.708286 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:11:59.708258 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zprxj_761f94f3-af3c-4bea-9122-40ba7a8a2ec0/payload-pre-processing/0.log" Jul 22 12:12:00.348253 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:00.348220 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qxjzw_91075186-e772-4761-9683-8e30ed4dcbb2/main/0.log" Jul 22 12:12:00.600659 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:00.600593 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-p5r59_a3988074-8ec4-49d6-a83e-e235d941e3cb/main/0.log" Jul 22 12:12:00.839140 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:00.839113 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4bp6g_6e05f0ea-63ab-4de4-8f50-1a7d51b10989/main/0.log" Jul 22 12:12:06.861163 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:06.861127 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:12:06.957455 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:06.957428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" limit=1024 Jul 22 12:12:12.082025 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:12.081992 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwjv7_d9c9f01e-fd6b-438e-a92c-11fddaaae685/global-pull-secret-syncer/0.log" Jul 22 12:12:12.184287 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:12.184258 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fq45z_365d1954-1ba1-4ad4-a394-243e9431a119/konnectivity-agent/0.log" Jul 22 12:12:12.339557 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:12.339493 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-40.ec2.internal_42c8e91a3838fc74d7daaaa54e385ed8/haproxy/0.log" Jul 22 12:12:18.066285 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:18.066256 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t2zlp_9165cfd9-1d39-455e-8cf0-e9c858320342/manager/0.log" Jul 22 12:12:20.230236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:20.230198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8bq6_8284171d-cec2-4e37-af8c-3edd06ebbfb7/node-exporter/0.log" Jul 22 12:12:20.252526 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:20.252500 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8bq6_8284171d-cec2-4e37-af8c-3edd06ebbfb7/kube-rbac-proxy/0.log" Jul 22 12:12:20.277718 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:20.277695 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8bq6_8284171d-cec2-4e37-af8c-3edd06ebbfb7/init-textfile/0.log" Jul 22 12:12:22.191124 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191085 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4"] Jul 22 12:12:22.191628 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191607 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" containerName="maas-api" Jul 22 12:12:22.191708 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191636 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" containerName="maas-api" Jul 22 12:12:22.191708 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191648 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" Jul 22 12:12:22.191708 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191657 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" Jul 22 12:12:22.191852 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191795 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75ef404-49f7-4534-8224-d11ce8dd3d37" containerName="istio-proxy" Jul 22 12:12:22.191852 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.191808 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d14f3297-3b62-4a63-b2ae-53ed7af4289b" containerName="maas-api" Jul 22 12:12:22.195074 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.195052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.197347 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.197327 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cxcj\"/\"kube-root-ca.crt\"" Jul 22 12:12:22.197818 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.197801 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cxcj\"/\"openshift-service-ca.crt\"" Jul 22 12:12:22.197818 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.197810 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5cxcj\"/\"default-dockercfg-c9d98\"" Jul 22 12:12:22.202099 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.202073 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4"] Jul 22 12:12:22.351543 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.351505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsw7\" (UniqueName: \"kubernetes.io/projected/8a36d4cd-e864-485a-860f-3e8b0f479a82-kube-api-access-9jsw7\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.351730 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.351564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-proc\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.351730 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.351636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-lib-modules\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.351730 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.351677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-podres\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.351730 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.351699 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-sys\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.452997 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.452866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-proc\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.452997 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.452987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-lib-modules\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453034 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-proc\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-podres\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-sys\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsw7\" (UniqueName: \"kubernetes.io/projected/8a36d4cd-e864-485a-860f-3e8b0f479a82-kube-api-access-9jsw7\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-podres\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-sys\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.453236 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.453181 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a36d4cd-e864-485a-860f-3e8b0f479a82-lib-modules\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.465776 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.465751 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsw7\" (UniqueName: \"kubernetes.io/projected/8a36d4cd-e864-485a-860f-3e8b0f479a82-kube-api-access-9jsw7\") pod \"perf-node-gather-daemonset-hd2g4\" (UID: \"8a36d4cd-e864-485a-860f-3e8b0f479a82\") " pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.505879 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.505838 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.648944 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.648910 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4"] Jul 22 12:12:22.652928 ip-10-0-136-40 kubenswrapper[2569]: W0722 12:12:22.652893 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a36d4cd_e864_485a_860f_3e8b0f479a82.slice/crio-722d9305d46bb99877211c25f299ffc0285a9c359062b5edcb406db007d392d1 WatchSource:0}: Error finding container 722d9305d46bb99877211c25f299ffc0285a9c359062b5edcb406db007d392d1: Status 404 returned error can't find the container with id 722d9305d46bb99877211c25f299ffc0285a9c359062b5edcb406db007d392d1 Jul 22 12:12:22.937520 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.937485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" event={"ID":"8a36d4cd-e864-485a-860f-3e8b0f479a82","Type":"ContainerStarted","Data":"d4644408dc2ccf4c93be7a4061412b3a72e1318972dec5f59be2431d1baa7969"} Jul 22 12:12:22.937690 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.937563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" event={"ID":"8a36d4cd-e864-485a-860f-3e8b0f479a82","Type":"ContainerStarted","Data":"722d9305d46bb99877211c25f299ffc0285a9c359062b5edcb406db007d392d1"} Jul 22 12:12:22.937690 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.937599 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:22.955433 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:22.955379 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" podStartSLOduration=0.955363625 podStartE2EDuration="955.363625ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:12:22.953198302 +0000 UTC m=+2913.114740997" watchObservedRunningTime="2026-07-22 12:12:22.955363625 +0000 UTC m=+2913.116906369" Jul 22 12:12:23.017743 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:23.017716 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/1.log" Jul 22 12:12:23.022723 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:23.022698 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gs8kc_c40e1c5a-dc8b-4702-a625-6761d7e68330/console-operator/2.log" Jul 22 12:12:23.563833 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:23.563799 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b68c87c5f-ghnw4_fb2ed0c0-76bd-4f92-8093-89d95d42c03b/console/0.log" Jul 22 12:12:24.220039 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:24.220007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-dkv9p_0344bcd5-3504-482c-af78-891468e3db88/volume-data-source-validator/0.log" Jul 22 12:12:25.423646 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:25.423611 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-psq2l_0150094a-8a0e-4152-a85f-481cdede5afa/dns/0.log" Jul 22 12:12:25.448984 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:25.448927 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-psq2l_0150094a-8a0e-4152-a85f-481cdede5afa/kube-rbac-proxy/0.log" Jul 22 12:12:25.478484 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:25.478457 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-579l5_299c4307-97a1-4299-b80f-9cda990540a2/dns-node-resolver/0.log" Jul 22 12:12:26.149220 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:26.149187 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6rxv_058ab9d2-04b1-4be5-9bfd-3d8c14900d49/node-ca/0.log" Jul 22 12:12:27.284163 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.284132 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dbmnmv_90e859e1-8863-422c-9a49-d8749e5b334f/istio-proxy/0.log" Jul 22 12:12:27.426456 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.426432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gdj7_68288dc1-7b9d-49d8-82e4-20b7d246e510/discovery/0.log" Jul 22 12:12:27.453108 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.453089 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ffd4cd5bc-fjtgs_370974f3-afe9-436b-9bca-a7fd3c84cd02/kube-auth-proxy/0.log" Jul 22 12:12:27.517903 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.517877 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/0.log" Jul 22 12:12:27.543082 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.543016 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/1.log" Jul 22 12:12:27.583032 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:27.583006 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zprxj_761f94f3-af3c-4bea-9122-40ba7a8a2ec0/payload-pre-processing/0.log" Jul 22 12:12:28.416478 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:28.416442 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-785x8_92b43cfb-ec0f-4f04-b4df-eb01a3105f06/serve-healthcheck-canary/0.log" Jul 22 12:12:28.952048 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:28.952023 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5cxcj/perf-node-gather-daemonset-hd2g4" Jul 22 12:12:29.008314 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:29.008294 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-g4rwm_96174ad0-30ee-46ab-9e18-8eafb79af050/insights-operator/0.log" Jul 22 12:12:29.008811 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:29.008792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-g4rwm_96174ad0-30ee-46ab-9e18-8eafb79af050/insights-operator/1.log" Jul 22 12:12:29.203891 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:29.203826 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5bw4_5455f9ae-4a87-468e-8025-adc4622ed566/kube-rbac-proxy/0.log" Jul 22 12:12:29.225941 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:29.225922 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5bw4_5455f9ae-4a87-468e-8025-adc4622ed566/exporter/0.log" Jul 22 12:12:29.250238 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:29.250222 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5bw4_5455f9ae-4a87-468e-8025-adc4622ed566/extractor/0.log" Jul 22 12:12:31.919374 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:31.919352 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74bd8b57c-9whxb_caa60025-11c9-437d-92b0-c982b1a21f37/manager/0.log" Jul 22 12:12:32.071161 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.071128 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m_6e9df6e6-bcda-4214-b8d1-13fedd123f54/manager/0.log" Jul 22 12:12:32.095070 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.095045 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74f4ff9cd9-2sf9m_6e9df6e6-bcda-4214-b8d1-13fedd123f54/copy-manifests/0.log" Jul 22 12:12:32.679139 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.679102 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dbmnmv_90e859e1-8863-422c-9a49-d8749e5b334f/istio-proxy/0.log" Jul 22 12:12:32.803480 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.803453 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gdj7_68288dc1-7b9d-49d8-82e4-20b7d246e510/discovery/0.log" Jul 22 12:12:32.815880 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.815853 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ffd4cd5bc-fjtgs_370974f3-afe9-436b-9bca-a7fd3c84cd02/kube-auth-proxy/0.log" Jul 22 12:12:32.853557 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.853533 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/0.log" Jul 22 12:12:32.879141 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.879118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/1.log" Jul 22 12:12:32.908717 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:32.908690 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zprxj_761f94f3-af3c-4bea-9122-40ba7a8a2ec0/payload-pre-processing/0.log" Jul 22 12:12:33.682365 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:33.682338 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-n98hc_2a4949a1-2b9d-4f55-9fd9-bf5ddc4f5048/openshift-lws-operator/0.log" Jul 22 12:12:34.238532 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.238503 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dbmnmv_90e859e1-8863-422c-9a49-d8749e5b334f/istio-proxy/0.log" Jul 22 12:12:34.363167 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.363140 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gdj7_68288dc1-7b9d-49d8-82e4-20b7d246e510/discovery/0.log" Jul 22 12:12:34.375825 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.375800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ffd4cd5bc-fjtgs_370974f3-afe9-436b-9bca-a7fd3c84cd02/kube-auth-proxy/0.log" Jul 22 12:12:34.419270 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.419231 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/0.log" Jul 22 12:12:34.444602 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.444582 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-zl9g6_d412bde1-a4b9-4024-9815-879a50c84655/istio-proxy/1.log" Jul 22 12:12:34.469089 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:34.469071 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zprxj_761f94f3-af3c-4bea-9122-40ba7a8a2ec0/payload-pre-processing/0.log" Jul 22 12:12:39.137207 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:39.137169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-d8z7t_22b92f2e-6199-4aea-85cc-33afb97f8ea0/kube-storage-version-migrator-operator/1.log" Jul 22 12:12:39.137984 ip-10-0-136-40 kubenswrapper[2569]: I0722 12:12:39.137964 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-d8z7t_22b92f2e-6199-4aea-85cc-33afb97f8ea0/kube-storage-version-migrator-operator/0.log"