Aug 04 19:15:38.919346 ip-10-0-140-250 systemd[1]: Starting Kubernetes Kubelet... Aug 04 19:15:39.295067 ip-10-0-140-250 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:39.295067 ip-10-0-140-250 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 19:15:39.295067 ip-10-0-140-250 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:39.295067 ip-10-0-140-250 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 19:15:39.295067 ip-10-0-140-250 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:39.295751 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.295128 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 19:15:39.298565 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298550 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.298565 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298566 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298570 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298573 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298576 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298579 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298581 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298584 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298587 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298590 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298593 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298595 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298598 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298601 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298603 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298606 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298609 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298611 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298614 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298618 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.298632 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298620 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298623 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298625 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298628 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298631 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298633 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298636 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298639 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298642 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298644 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298647 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298650 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298652 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298655 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298657 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298660 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298663 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298666 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298669 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298672 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.299096 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298674 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298677 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298681 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298684 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298687 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298691 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298695 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298699 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298702 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298704 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298707 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298710 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298712 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298715 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298718 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298721 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298724 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298727 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298729 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298732 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.299581 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298734 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298737 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298740 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298742 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298745 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298748 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298750 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298753 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298755 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298758 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298761 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298764 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298766 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298771 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298773 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298776 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298778 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298781 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298783 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298786 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.300115 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298789 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298791 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298794 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298796 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298799 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298802 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298807 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298810 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298813 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298816 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298819 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298822 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298825 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298828 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298830 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298833 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298836 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298838 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.300593 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.298842 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.301544 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301531 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.301544 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301543 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301547 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301551 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301554 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301563 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301566 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301568 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301571 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301574 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301576 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301579 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301581 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301584 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301586 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301589 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301591 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301594 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301597 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301599 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301602 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.301612 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301604 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301607 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301610 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301614 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301617 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301620 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301623 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301626 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301629 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301632 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301634 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301636 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301646 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301656 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301659 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301662 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301665 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301673 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301675 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.302172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301678 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301680 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301683 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301685 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301688 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301690 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301693 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301695 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301698 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301700 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301703 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301707 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301709 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301712 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301718 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301722 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301724 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301727 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301730 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301732 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.302678 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301735 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301737 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301740 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301743 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301745 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301748 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301751 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301753 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301756 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301758 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301762 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301771 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301774 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301777 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301780 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301784 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301788 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301791 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301794 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.303211 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301797 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301800 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301802 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301805 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301807 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301810 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301812 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301816 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301818 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301821 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301824 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301826 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301829 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301832 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301834 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301836 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301839 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301841 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301844 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.301847 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.303664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302307 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302313 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302317 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302321 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302329 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302332 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302335 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302337 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302340 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302342 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302345 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302348 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302350 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302353 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302356 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302358 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302362 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302366 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302368 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.304189 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302371 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302374 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302377 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302379 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302382 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302385 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302387 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302390 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302392 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302395 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302397 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302400 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302402 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302405 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302407 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302411 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302414 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302417 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302425 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302428 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.304684 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302430 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302433 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302436 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302438 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302441 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302444 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302462 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302466 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302469 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302471 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302474 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302477 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302479 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302483 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302485 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302488 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302491 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302494 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302496 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302499 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.305178 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302501 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302504 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302506 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302509 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302511 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302514 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302517 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302519 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302523 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302527 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302530 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302540 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302544 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302546 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302549 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302552 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302555 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302557 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302560 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302563 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.305664 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302565 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302568 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302570 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302573 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302575 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302577 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302580 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302582 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302585 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302587 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302589 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302592 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302594 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302597 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302599 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302602 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302604 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302607 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302609 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.302612 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.306165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302675 2567 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302682 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302691 2567 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302696 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302715 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302718 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302722 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302726 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302729 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302732 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302736 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302739 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302742 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302745 2567 flags.go:64] FLAG: --cgroup-root="" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302748 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302751 2567 flags.go:64] FLAG: --client-ca-file="" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302754 2567 flags.go:64] FLAG: --cloud-provider="external" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302757 2567 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302763 2567 flags.go:64] FLAG: --cluster-domain="" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302766 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302769 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302772 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302776 2567 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302779 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 19:15:39.306660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302783 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302786 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302789 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302792 2567 flags.go:64] FLAG: --contention-profiling="false" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302795 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302798 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302801 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302807 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302811 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302814 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302817 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302820 2567 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302823 2567 flags.go:64] FLAG: --enable-server="true" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302832 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302837 2567 flags.go:64] FLAG: --event-burst="100" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302840 2567 flags.go:64] FLAG: --event-qps="50" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302843 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302846 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302849 2567 flags.go:64] FLAG: --eviction-hard="" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302853 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302855 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302858 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302861 2567 flags.go:64] FLAG: --eviction-soft="" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302864 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302867 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 19:15:39.307300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302870 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302873 2567 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302876 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302878 2567 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302881 2567 flags.go:64] FLAG: --feature-gates="" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302885 2567 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302887 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302891 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302894 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302896 2567 flags.go:64] FLAG: --healthz-port="10248" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302899 2567 flags.go:64] FLAG: --help="false" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302902 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302905 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302908 2567 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302912 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302916 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302920 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302923 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302926 2567 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302928 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302931 2567 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302935 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302938 2567 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302941 2567 flags.go:64] FLAG: --kube-reserved="" Aug 04 19:15:39.307946 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302944 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302946 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302949 2567 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302952 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302955 2567 flags.go:64] FLAG: --lock-file="" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302958 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302960 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302963 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302968 2567 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302971 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302973 2567 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302976 2567 flags.go:64] FLAG: --logging-format="text" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302979 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302982 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302985 2567 flags.go:64] FLAG: --manifest-url="" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302988 2567 flags.go:64] FLAG: --manifest-url-header="" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302996 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.302999 2567 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303003 2567 flags.go:64] FLAG: --max-pods="110" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303006 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303009 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303012 2567 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303016 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303019 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303023 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 19:15:39.308538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303026 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303033 2567 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303036 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303039 2567 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303042 2567 flags.go:64] FLAG: --pod-cidr="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303046 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303063 2567 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303066 2567 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303070 2567 flags.go:64] FLAG: --pods-per-core="0" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303073 2567 flags.go:64] FLAG: --port="10250" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303076 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303079 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07ed63478bb87c2a3" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303082 2567 flags.go:64] FLAG: --qos-reserved="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303085 2567 flags.go:64] FLAG: --read-only-port="10255" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303088 2567 flags.go:64] FLAG: --register-node="true" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303090 2567 flags.go:64] FLAG: --register-with-taints="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303094 2567 flags.go:64] FLAG: --registry-burst="10" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303097 2567 flags.go:64] FLAG: --registry-qps="5" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303100 2567 flags.go:64] FLAG: --reserved-cpus="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303103 2567 flags.go:64] FLAG: --reserved-memory="" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303106 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303109 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303112 2567 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303115 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303118 2567 flags.go:64] FLAG: --runonce="false" Aug 04 19:15:39.309192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303121 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303124 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303127 2567 flags.go:64] FLAG: --seccomp-default="false" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303130 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303132 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303137 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303141 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303144 2567 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303147 2567 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303150 2567 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303153 2567 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303156 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303159 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303162 2567 flags.go:64] FLAG: --system-cgroups="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303165 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303170 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303173 2567 flags.go:64] FLAG: --tls-cert-file="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303176 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303179 2567 flags.go:64] FLAG: --tls-min-version="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303182 2567 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303185 2567 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303188 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303191 2567 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303194 2567 flags.go:64] FLAG: --v="2" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303198 2567 flags.go:64] FLAG: --version="false" Aug 04 19:15:39.309816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303202 2567 flags.go:64] FLAG: --vmodule="" Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303205 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.303208 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303337 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303341 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303344 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303347 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303349 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303352 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303355 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303357 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303360 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303364 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303368 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303370 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303373 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303376 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303379 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303382 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303384 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.310447 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303387 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303390 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303393 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303396 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303398 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303402 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303405 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303408 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303410 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303413 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303415 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303418 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303420 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303423 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303425 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303428 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303431 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303433 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303437 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.310930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303440 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303443 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303446 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303449 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303451 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303456 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303460 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303462 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303465 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303468 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303471 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303473 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303476 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303479 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303482 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303485 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303487 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303490 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303492 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303495 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.311405 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303497 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303500 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303502 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303504 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303507 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303510 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303512 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303515 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303517 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303520 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303522 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303525 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303527 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303530 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303533 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303535 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303538 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303542 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303545 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303548 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.312004 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303550 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303553 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303556 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303558 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303561 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303563 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303566 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303569 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303572 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303576 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303579 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303582 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303585 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303588 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303591 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303594 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303597 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303599 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303602 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.312688 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303605 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303607 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303610 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.303612 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.304115 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.310469 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.310485 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.310503 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.310511 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310565 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310571 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310574 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310577 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310580 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310583 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.313516 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310586 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310589 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310591 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310594 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310597 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310599 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310602 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310605 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310608 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310610 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310613 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310617 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310621 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310624 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310626 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310629 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310632 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310634 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310637 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310639 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.314186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310642 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310645 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310648 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310652 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310655 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310658 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310660 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310663 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310666 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310668 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310671 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310674 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310676 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310679 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310682 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310684 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310687 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310689 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310692 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310694 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.314813 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310697 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310700 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310702 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310705 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310707 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310710 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310712 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310715 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310717 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310720 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310722 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310725 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310727 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310730 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310733 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310736 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310739 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310742 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310745 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310748 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.315317 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310750 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310753 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310756 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310758 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310761 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310765 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310769 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310771 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310774 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310776 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310779 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310781 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310784 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310786 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310790 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310794 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310797 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310800 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310803 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.315803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310805 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310808 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310811 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310814 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310816 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310819 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310822 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310824 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310828 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310831 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310833 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310836 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310839 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310841 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.310847 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:39.316346 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310976 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310981 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310984 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310987 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310989 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310992 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310994 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310997 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.310999 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311002 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311004 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311007 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311010 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311013 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311016 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311020 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311023 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311025 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311028 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311031 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:39.316714 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311033 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311036 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311038 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311040 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311043 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311060 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311064 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311066 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311069 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311072 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311075 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311077 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311080 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311082 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311085 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311087 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311090 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311092 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311095 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311097 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:39.317473 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311100 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311102 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311105 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311107 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311110 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311112 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311115 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311119 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311122 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311126 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311130 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311132 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311135 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311138 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311141 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311143 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311146 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311149 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311152 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:39.318235 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311155 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311157 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311160 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311163 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311166 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311169 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311171 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311174 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311177 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311180 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311183 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311193 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311196 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311199 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311201 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311204 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311207 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311209 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311212 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:39.318686 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311214 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311217 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311220 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311222 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311225 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311227 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311230 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311232 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311235 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311237 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311240 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311243 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311245 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311248 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311250 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311253 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311256 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311259 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311261 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311264 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:39.319301 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:39.311266 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.311271 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.311364 2567 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.315684 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.316846 2567 server.go:1019] "Starting client certificate rotation" Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.316954 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.317020 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:39.319773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.319204 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:39.320300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.320280 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:39.330708 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.330690 2567 log.go:25] "Validated CRI v1 runtime API" Aug 04 19:15:39.335013 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.334996 2567 log.go:25] "Validated CRI v1 image API" Aug 04 19:15:39.336521 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.336503 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 19:15:39.338705 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.338688 2567 fs.go:135] Filesystem UUIDs: map[1a70ba82-c09c-4f3a-9ae4-b3f69b958392:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 98730310-6262-4652-92a6-986462075880:/dev/nvme0n1p3] Aug 04 19:15:39.338758 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.338706 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 19:15:39.345007 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.344890 2567 manager.go:216] Machine: {Timestamp:2026-08-04 19:15:39.343274581 +0000 UTC m=+0.313301090 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130384 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a1a79671c579eb1bf93d56284a38 SystemUUID:ec20a1a7-9671-c579-eb1b-f93d56284a38 BootID:9d595543-d9c4-4da4-a42f-341385486442 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:38:2b:8a:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:38:2b:8a:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:f4:33:f4:28:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 19:15:39.345007 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.345002 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 19:15:39.345126 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.345091 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 19:15:39.347994 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.347976 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:39.348106 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.348081 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 19:15:39.348232 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.348110 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 19:15:39.348872 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.348862 2567 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 19:15:39.348901 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.348874 2567 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 19:15:39.348901 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.348882 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:39.349746 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.349735 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:39.350521 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.350510 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 19:15:39.354037 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354026 2567 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:39.354169 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354159 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 19:15:39.354197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354192 2567 kubelet.go:486] "Attempting to sync node with API server" Aug 04 19:15:39.354230 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354209 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 19:15:39.354230 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354222 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 19:15:39.354284 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354231 2567 kubelet.go:398] "Adding apiserver pod source" Aug 04 19:15:39.354284 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354240 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 19:15:39.354625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354607 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:39.354625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.354628 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:39.357676 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.357651 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 19:15:39.359065 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.359032 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 19:15:39.359156 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.359088 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 19:15:39.360270 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360257 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360276 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360284 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360292 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360304 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360312 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360323 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360331 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360341 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 19:15:39.360346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360350 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 19:15:39.360721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360361 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 19:15:39.360721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360370 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 19:15:39.360721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360396 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 19:15:39.360721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.360403 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 19:15:39.363421 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.363407 2567 server.go:1268] "Started kubelet" Aug 04 19:15:39.363667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.363642 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 19:15:39.363726 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.363687 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 19:15:39.364179 ip-10-0-140-250 systemd[1]: Started Kubernetes Kubelet. Aug 04 19:15:39.364367 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.364186 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 19:15:39.365009 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.364951 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 19:15:39.365921 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.365901 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 19:15:39.366015 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.365951 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 19:15:39.366120 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.366100 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 19:15:39.366349 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.366331 2567 server.go:310] "Adding debug handlers to kubelet server" Aug 04 19:15:39.368218 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.368203 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 19:15:39.369435 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.369417 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 19:15:39.369435 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.369431 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:39.369574 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.369538 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.370185 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370138 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 19:15:39.370185 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370152 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 19:15:39.370185 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370157 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 19:15:39.370353 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370254 2567 reconstruct.go:97] "Volume reconstruction finished" Aug 04 19:15:39.370353 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370263 2567 reconciler.go:29] "Reconciler: start to sync state" Aug 04 19:15:39.370554 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370538 2567 factory.go:153] Registering CRI-O factory Aug 04 19:15:39.370631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370558 2567 factory.go:223] Registration of the crio container factory successfully Aug 04 19:15:39.370684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370644 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 19:15:39.370684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370655 2567 factory.go:55] Registering systemd factory Aug 04 19:15:39.370684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370664 2567 factory.go:223] Registration of the systemd container factory successfully Aug 04 19:15:39.370684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370675 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9ldf" Aug 04 19:15:39.370684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370683 2567 factory.go:103] Registering Raw factory Aug 04 19:15:39.370914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.370709 2567 manager.go:1196] Started watching for new ooms in manager Aug 04 19:15:39.371142 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.371127 2567 manager.go:349] Starting recovery of all containers Aug 04 19:15:39.371942 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.371916 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 19:15:39.372074 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.372034 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 19:15:39.373030 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.372116 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-250.ec2.internal.18c8b0284b7294e9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-250.ec2.internal,UID:ip-10-0-140-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-250.ec2.internal,},FirstTimestamp:2026-08-04 19:15:39.363382505 +0000 UTC m=+0.333409015,LastTimestamp:2026-08-04 19:15:39.363382505 +0000 UTC m=+0.333409015,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-250.ec2.internal,}" Aug 04 19:15:39.378782 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.378691 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9ldf" Aug 04 19:15:39.383339 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.383322 2567 manager.go:354] Recovery completed Aug 04 19:15:39.387594 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.387582 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.389331 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389310 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.389403 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389346 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.389403 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389360 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.389847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389833 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 19:15:39.389847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389843 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 19:15:39.389954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389854 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.389954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.389861 2567 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:39.390519 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.390504 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.390597 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.390534 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.390597 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.390548 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.391179 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.391165 2567 policy_none.go:49] "None policy: Start" Aug 04 19:15:39.391179 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.391180 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 19:15:39.391276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.391190 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 19:15:39.391407 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.391336 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-250.ec2.internal.18c8b0284cfe7cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-250.ec2.internal,UID:ip-10-0-140-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-250.ec2.internal,},FirstTimestamp:2026-08-04 19:15:39.389328602 +0000 UTC m=+0.359355117,LastTimestamp:2026-08-04 19:15:39.389328602 +0000 UTC m=+0.359355117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-250.ec2.internal,}" Aug 04 19:15:39.392238 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.392228 2567 policy_none.go:47] "Start" Aug 04 19:15:39.420662 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.420642 2567 manager.go:337] "Starting Device Plugin manager" Aug 04 19:15:39.420744 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.420673 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 19:15:39.420744 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.420685 2567 server.go:87] "Starting device plugin registration server" Aug 04 19:15:39.420924 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.420911 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 19:15:39.420967 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.420925 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 19:15:39.421039 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.421021 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 19:15:39.421131 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.421120 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 19:15:39.421182 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.421132 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 19:15:39.421530 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.421513 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 19:15:39.421578 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.421564 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.496776 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.496741 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 19:15:39.497914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.497900 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 19:15:39.497972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.497934 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 19:15:39.497972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.497958 2567 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 19:15:39.498042 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.497998 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 19:15:39.500496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.500476 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:39.521550 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.521534 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.522309 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.522290 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.522367 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.522321 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.522367 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.522332 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.522367 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.522355 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.528209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.528196 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.528257 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.528215 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-250.ec2.internal\": node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.541268 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.541251 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.598960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.598937 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal"] Aug 04 19:15:39.599040 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.599003 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.600559 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.600542 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.600634 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.600572 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.600634 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.600582 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.601799 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.601787 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.601906 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.601891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.601940 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.601926 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.602599 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602584 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.602672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602614 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.602672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602624 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.602672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602585 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.602765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602686 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.602765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.602698 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.603830 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.603817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.603874 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.603841 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:39.604545 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.604528 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:39.604637 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.604558 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:39.604637 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.604573 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:39.633873 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.633852 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-250.ec2.internal\" not found" node="ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.638565 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.638548 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-250.ec2.internal\" not found" node="ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.641627 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.641612 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.672006 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.671982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.672006 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.672006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.672179 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.672022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffeedbcd8e5be3b25f752d90133cc1ac-config\") pod \"kube-apiserver-proxy-ip-10-0-140-250.ec2.internal\" (UID: \"ffeedbcd8e5be3b25f752d90133cc1ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.742091 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.742046 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.772342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772318 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.772428 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.772428 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffeedbcd8e5be3b25f752d90133cc1ac-config\") pod \"kube-apiserver-proxy-ip-10-0-140-250.ec2.internal\" (UID: \"ffeedbcd8e5be3b25f752d90133cc1ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.772522 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.772522 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffeedbcd8e5be3b25f752d90133cc1ac-config\") pod \"kube-apiserver-proxy-ip-10-0-140-250.ec2.internal\" (UID: \"ffeedbcd8e5be3b25f752d90133cc1ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.772522 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.772421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f3042af8b8ad801d4b3bcf93179fca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal\" (UID: \"7f3042af8b8ad801d4b3bcf93179fca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.842821 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.842759 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:39.937985 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.937938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.941097 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:39.941076 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:39.943362 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:39.943346 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.044772 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.044554 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.145612 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.145544 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.246346 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.246295 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.316404 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.316367 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 19:15:40.316860 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.316489 2567 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.140.250:45000->18.214.138.163:6443: use of closed network connection" Aug 04 19:15:40.316860 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.316515 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 19:15:40.316860 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.316559 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.140.250:45000->18.214.138.163:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 19:15:40.347470 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.347431 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.383369 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.383324 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 19:10:39 +0000 UTC" deadline="2028-02-29 09:16:29.750057977 +0000 UTC" Aug 04 19:15:40.383369 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.383361 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13766h0m49.366700063s" Aug 04 19:15:40.395399 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.395382 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:40.447965 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:40.447932 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-250.ec2.internal\" not found" Aug 04 19:15:40.505729 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.505695 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:40.532636 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:40.532598 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3042af8b8ad801d4b3bcf93179fca3.slice/crio-194908e7f9353fa4de373057c481ffbe8e29803f645a9008bc77d7a7eeb4bff4 WatchSource:0}: Error finding container 194908e7f9353fa4de373057c481ffbe8e29803f645a9008bc77d7a7eeb4bff4: Status 404 returned error can't find the container with id 194908e7f9353fa4de373057c481ffbe8e29803f645a9008bc77d7a7eeb4bff4 Aug 04 19:15:40.536276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.536258 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:15:40.570089 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.570044 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" Aug 04 19:15:40.582234 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.582207 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:40.583925 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.583912 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" Aug 04 19:15:40.594010 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:40.593988 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:41.165980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.165949 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:41.355685 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.355656 2567 apiserver.go:52] "Watching apiserver" Aug 04 19:15:41.364619 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.364593 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 19:15:41.366418 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.366399 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hswfg","openshift-multus/network-metrics-daemon-2ldng","openshift-network-diagnostics/network-check-target-62hbz","kube-system/global-pull-secret-syncer-b4hwg","kube-system/konnectivity-agent-7tlx9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp","openshift-image-registry/node-ca-k46fv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal","openshift-network-operator/iptables-alerter-679gz","openshift-ovn-kubernetes/ovnkube-node-mfpxw","kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kvzzm","openshift-dns/node-resolver-pwb2t","openshift-multus/multus-additional-cni-plugins-q5k47"] Aug 04 19:15:41.367728 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.367713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.368747 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.368729 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.369807 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.369787 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:41.369894 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.369871 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:41.370338 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370320 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.370411 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370339 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwvdr\"" Aug 04 19:15:41.370411 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370340 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.370509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370326 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 19:15:41.370561 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370547 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:41.370690 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370670 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.370785 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.370740 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:41.370901 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370882 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntkm8\"" Aug 04 19:15:41.370901 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.370892 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.371282 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.371265 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.371920 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.371899 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.373283 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.373267 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.374293 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.374278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 19:15:41.374387 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.374371 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4s5dp\"" Aug 04 19:15:41.374534 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.374519 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 19:15:41.374608 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.374540 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.375704 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.376071 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.376431 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.376840 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.376871 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.376945 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.377087 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7kpm\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.377115 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lc8tl\"" Aug 04 19:15:41.377333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.377198 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.378739 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.378718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.378833 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.378780 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:41.379284 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.379266 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.379574 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.379561 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 19:15:41.379640 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.379612 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.379924 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.379907 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 19:15:41.379924 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.379919 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-swlrh\"" Aug 04 19:15:41.380208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380196 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.380471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b5c4ce2-5922-41bb-8802-f0b554e441b4-host\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.380471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b5c4ce2-5922-41bb-8802-f0b554e441b4-serviceca\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.380471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:41.380635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-kubelet-config\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.380635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.380635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbvf\" (UniqueName: \"kubernetes.io/projected/0b5c4ce2-5922-41bb-8802-f0b554e441b4-kube-api-access-wdbvf\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.380635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380583 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11e729b-8b85-4b6a-a86f-3a58fb19471e-hosts-file\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.380635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11e729b-8b85-4b6a-a86f-3a58fb19471e-tmp-dir\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.380826 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chrt\" (UniqueName: \"kubernetes.io/projected/c11e729b-8b85-4b6a-a86f-3a58fb19471e-kube-api-access-9chrt\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.380826 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-dbus\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.380959 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.380945 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:41.381685 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.381535 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.382396 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.382378 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 19:15:41.382396 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.382389 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 19:15:41.382551 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.382417 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zn78z\"" Aug 04 19:15:41.382551 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.382419 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 19:15:41.382759 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.382746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.383438 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.383419 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 19:15:41.383524 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.383460 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.383524 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.383485 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.383893 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.383876 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 19:15:41.383952 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.383937 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 19:15:41.385114 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.385094 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrqkv\"" Aug 04 19:15:41.385114 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.385112 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:41.385387 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.385371 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:41.385477 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.385420 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmh2g\"" Aug 04 19:15:41.404579 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.404557 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzs6z" Aug 04 19:15:41.412302 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.412284 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzs6z" Aug 04 19:15:41.473128 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.473076 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 19:15:41.481376 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-system-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-socket-dir-parent\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481398 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-kubelet\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-systemd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-ovn\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.481510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-node-log\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cnibin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-bin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-var-lib-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-sys\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481628 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b5c4ce2-5922-41bb-8802-f0b554e441b4-host\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481673 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-multus\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc83340-7d5d-4d02-8a45-57e8242471f6-host-slash\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.481760 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b5c4ce2-5922-41bb-8802-f0b554e441b4-host\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-env-overrides\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-tmp\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chrt\" (UniqueName: \"kubernetes.io/projected/c11e729b-8b85-4b6a-a86f-3a58fb19471e-kube-api-access-9chrt\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.481945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-dbus\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-os-release\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-log-socket\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-var-lib-kubelet\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482176 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-dbus\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.482192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cnibin\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-os-release\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc83340-7d5d-4d02-8a45-57e8242471f6-iptables-alerter-script\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482326 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-etc-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-netd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-agent-certs\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482391 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-script-lib\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysconfig\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-conf\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-tuned\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z859\" (UniqueName: \"kubernetes.io/projected/d2912b55-9498-4cc6-ab05-fb8475e1deb2-kube-api-access-9z859\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gx7\" (UniqueName: \"kubernetes.io/projected/acc83340-7d5d-4d02-8a45-57e8242471f6-kube-api-access-g8gx7\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-systemd-units\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.482589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-config\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-kubernetes\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-run\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-daemon-config\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-multus-certs\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482741 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-system-cni-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-modprobe-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482787 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkqq\" (UniqueName: \"kubernetes.io/projected/ded3861d-b5e2-484f-81d7-db61175db4e4-kube-api-access-7bkqq\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-netns\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482837 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11e729b-8b85-4b6a-a86f-3a58fb19471e-hosts-file\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482894 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-device-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11e729b-8b85-4b6a-a86f-3a58fb19471e-hosts-file\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482919 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dfxx\" (UniqueName: \"kubernetes.io/projected/ba41eb7a-3667-4dee-b319-281fbafe2993-kube-api-access-4dfxx\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.483226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b5c4ce2-5922-41bb-8802-f0b554e441b4-serviceca\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.482986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc9k\" (UniqueName: \"kubernetes.io/projected/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-kube-api-access-2mc9k\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-bin\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba41eb7a-3667-4dee-b319-281fbafe2993-ovn-node-metrics-cert\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-k8s-cni-cncf-io\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-etc-kubernetes\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-kubelet\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-host\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.483258 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-hostroot\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.483370 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:41.983332082 +0000 UTC m=+2.953358600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:41.484021 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483439 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b5c4ce2-5922-41bb-8802-f0b554e441b4-serviceca\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-systemd\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-lib-modules\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-kubelet-config\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-kubelet-config\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cni-binary-copy\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-slash\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7wm\" (UniqueName: \"kubernetes.io/projected/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-kube-api-access-mz7wm\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-konnectivity-ca\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-netns\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbvf\" (UniqueName: \"kubernetes.io/projected/0b5c4ce2-5922-41bb-8802-f0b554e441b4-kube-api-access-wdbvf\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11e729b-8b85-4b6a-a86f-3a58fb19471e-tmp-dir\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483806 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ql7v\" (UniqueName: \"kubernetes.io/projected/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kube-api-access-5ql7v\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.483820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-conf-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.484623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.484127 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11e729b-8b85-4b6a-a86f-3a58fb19471e-tmp-dir\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.495435 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.495408 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 19:15:41.495577 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.495471 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:41.495577 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.495490 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:41.495577 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.495503 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:41.495577 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.495572 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:41.99555614 +0000 UTC m=+2.965582661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:41.498646 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.498617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chrt\" (UniqueName: \"kubernetes.io/projected/c11e729b-8b85-4b6a-a86f-3a58fb19471e-kube-api-access-9chrt\") pod \"node-resolver-pwb2t\" (UID: \"c11e729b-8b85-4b6a-a86f-3a58fb19471e\") " pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.498749 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.498734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbvf\" (UniqueName: \"kubernetes.io/projected/0b5c4ce2-5922-41bb-8802-f0b554e441b4-kube-api-access-wdbvf\") pod \"node-ca-k46fv\" (UID: \"0b5c4ce2-5922-41bb-8802-f0b554e441b4\") " pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.502701 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.502665 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" event={"ID":"7f3042af8b8ad801d4b3bcf93179fca3","Type":"ContainerStarted","Data":"194908e7f9353fa4de373057c481ffbe8e29803f645a9008bc77d7a7eeb4bff4"} Aug 04 19:15:41.584797 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-hostroot\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-hostroot\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.584984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.584991 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-systemd\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585016 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-lib-modules\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-systemd\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cni-binary-copy\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585087 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-slash\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7wm\" (UniqueName: \"kubernetes.io/projected/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-kube-api-access-mz7wm\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-konnectivity-ca\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-netns\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-slash\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585194 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-lib-modules\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585242 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-netns\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585350 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585382 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ql7v\" (UniqueName: \"kubernetes.io/projected/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kube-api-access-5ql7v\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-conf-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-system-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-conf-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-system-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-socket-dir-parent\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585582 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-kubelet\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-socket-dir-parent\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-systemd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-kubelet\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-ovn\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585681 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cni-binary-copy\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-ovn\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.585838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-run-systemd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-node-log\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-konnectivity-ca\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cnibin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-bin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.585836 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-var-lib-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-sys\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-bin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585878 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-cnibin\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585835 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-node-log\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.585899 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:42.085881953 +0000 UTC m=+3.055908451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585891 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-var-lib-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585947 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-multus\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585924 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-sys\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc83340-7d5d-4d02-8a45-57e8242471f6-host-slash\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.585998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-env-overrides\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.586642 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-tmp\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586036 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-os-release\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-log-socket\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-var-lib-cni-multus\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-var-lib-kubelet\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586062 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc83340-7d5d-4d02-8a45-57e8242471f6-host-slash\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-os-release\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-log-socket\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586153 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-var-lib-kubelet\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cnibin\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-os-release\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cnibin\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-os-release\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.587544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc83340-7d5d-4d02-8a45-57e8242471f6-iptables-alerter-script\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-etc-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-netd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-agent-certs\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-script-lib\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysconfig\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-env-overrides\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-conf\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586492 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-tuned\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z859\" (UniqueName: \"kubernetes.io/projected/d2912b55-9498-4cc6-ab05-fb8475e1deb2-kube-api-access-9z859\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gx7\" (UniqueName: \"kubernetes.io/projected/acc83340-7d5d-4d02-8a45-57e8242471f6-kube-api-access-g8gx7\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586575 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-systemd-units\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-config\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-kubernetes\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.588342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-etc-openvswitch\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-run\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-daemon-config\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-netd\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-multus-certs\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-systemd-units\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586793 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-kubernetes\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc83340-7d5d-4d02-8a45-57e8242471f6-iptables-alerter-script\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586833 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-run\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysconfig\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586870 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-system-cni-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-modprobe-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586964 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkqq\" (UniqueName: \"kubernetes.io/projected/ded3861d-b5e2-484f-81d7-db61175db4e4-kube-api-access-7bkqq\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.586990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-netns\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.589199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-conf\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-system-cni-dir\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587105 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-multus-certs\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-device-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dfxx\" (UniqueName: \"kubernetes.io/projected/ba41eb7a-3667-4dee-b319-281fbafe2993-kube-api-access-4dfxx\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-sysctl-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587237 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc9k\" (UniqueName: \"kubernetes.io/projected/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-kube-api-access-2mc9k\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-bin\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba41eb7a-3667-4dee-b319-281fbafe2993-ovn-node-metrics-cert\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-k8s-cni-cncf-io\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-modprobe-d\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-cni-bin\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-etc-kubernetes\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-host-run-k8s-cni-cncf-io\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590164 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-cni-dir\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587511 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-device-dir\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-etc-kubernetes\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-run-netns\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-script-lib\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-kubelet\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-host\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ded3861d-b5e2-484f-81d7-db61175db4e4-host\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba41eb7a-3667-4dee-b319-281fbafe2993-host-kubelet\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587856 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba41eb7a-3667-4dee-b319-281fbafe2993-ovnkube-config\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.587896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-multus-daemon-config\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.588796 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-tmp\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.589707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ded3861d-b5e2-484f-81d7-db61175db4e4-etc-tuned\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.589844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338-agent-certs\") pod \"konnectivity-agent-7tlx9\" (UID: \"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338\") " pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.590992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.589904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba41eb7a-3667-4dee-b319-281fbafe2993-ovn-node-metrics-cert\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.594851 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.594445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ql7v\" (UniqueName: \"kubernetes.io/projected/b2e719c9-0d3b-471e-a451-d8c6a182a7e9-kube-api-access-5ql7v\") pod \"aws-ebs-csi-driver-node-mjnmp\" (UID: \"b2e719c9-0d3b-471e-a451-d8c6a182a7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.594851 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.594810 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gx7\" (UniqueName: \"kubernetes.io/projected/acc83340-7d5d-4d02-8a45-57e8242471f6-kube-api-access-g8gx7\") pod \"iptables-alerter-679gz\" (UID: \"acc83340-7d5d-4d02-8a45-57e8242471f6\") " pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.595000 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.594860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7wm\" (UniqueName: \"kubernetes.io/projected/5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7-kube-api-access-mz7wm\") pod \"multus-additional-cni-plugins-q5k47\" (UID: \"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7\") " pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.596398 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.596377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z859\" (UniqueName: \"kubernetes.io/projected/d2912b55-9498-4cc6-ab05-fb8475e1deb2-kube-api-access-9z859\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:41.596398 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.596389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dfxx\" (UniqueName: \"kubernetes.io/projected/ba41eb7a-3667-4dee-b319-281fbafe2993-kube-api-access-4dfxx\") pod \"ovnkube-node-mfpxw\" (UID: \"ba41eb7a-3667-4dee-b319-281fbafe2993\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.596597 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.596579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc9k\" (UniqueName: \"kubernetes.io/projected/8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70-kube-api-access-2mc9k\") pod \"multus-hswfg\" (UID: \"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70\") " pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.596718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.596696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkqq\" (UniqueName: \"kubernetes.io/projected/ded3861d-b5e2-484f-81d7-db61175db4e4-kube-api-access-7bkqq\") pod \"tuned-kvzzm\" (UID: \"ded3861d-b5e2-484f-81d7-db61175db4e4\") " pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.677979 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.677897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k46fv" Aug 04 19:15:41.685129 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.685096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwb2t" Aug 04 19:15:41.692955 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.692930 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:15:41.696546 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.696526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" Aug 04 19:15:41.704155 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.704137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-679gz" Aug 04 19:15:41.710859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.710836 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hswfg" Aug 04 19:15:41.717564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.717530 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:15:41.724474 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.724454 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5k47" Aug 04 19:15:41.732369 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.732350 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" Aug 04 19:15:41.776285 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.776246 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8749a6bd_aed8_4cd7_bd7d_51f1e2ae9c70.slice/crio-565815a5ebe727c8661d5efd5503c62ae75a975bd6c9142bf818212cbbb79479 WatchSource:0}: Error finding container 565815a5ebe727c8661d5efd5503c62ae75a975bd6c9142bf818212cbbb79479: Status 404 returned error can't find the container with id 565815a5ebe727c8661d5efd5503c62ae75a975bd6c9142bf818212cbbb79479 Aug 04 19:15:41.781997 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.781974 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded3861d_b5e2_484f_81d7_db61175db4e4.slice/crio-75f998e6cf59c8f2f3fbd4e7a176ba0fe61e92872e7f2f56dc7a5143aec46f82 WatchSource:0}: Error finding container 75f998e6cf59c8f2f3fbd4e7a176ba0fe61e92872e7f2f56dc7a5143aec46f82: Status 404 returned error can't find the container with id 75f998e6cf59c8f2f3fbd4e7a176ba0fe61e92872e7f2f56dc7a5143aec46f82 Aug 04 19:15:41.782730 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.782706 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba41eb7a_3667_4dee_b319_281fbafe2993.slice/crio-5a243fc1cc097e6c8a5b31be9931a8580877a463d3e65fda12e015bb342437a1 WatchSource:0}: Error finding container 5a243fc1cc097e6c8a5b31be9931a8580877a463d3e65fda12e015bb342437a1: Status 404 returned error can't find the container with id 5a243fc1cc097e6c8a5b31be9931a8580877a463d3e65fda12e015bb342437a1 Aug 04 19:15:41.783431 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.783399 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e719c9_0d3b_471e_a451_d8c6a182a7e9.slice/crio-b99de7109090e4a4e7275a236d92a7bffff3cd6f3eac7f5049ac0233207517bc WatchSource:0}: Error finding container b99de7109090e4a4e7275a236d92a7bffff3cd6f3eac7f5049ac0233207517bc: Status 404 returned error can't find the container with id b99de7109090e4a4e7275a236d92a7bffff3cd6f3eac7f5049ac0233207517bc Aug 04 19:15:41.784215 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.784174 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc83340_7d5d_4d02_8a45_57e8242471f6.slice/crio-ffe3a17aa866c6a22cbe5ce87fa183586056dfc95c0dd32a7343a4f1402ca4fc WatchSource:0}: Error finding container ffe3a17aa866c6a22cbe5ce87fa183586056dfc95c0dd32a7343a4f1402ca4fc: Status 404 returned error can't find the container with id ffe3a17aa866c6a22cbe5ce87fa183586056dfc95c0dd32a7343a4f1402ca4fc Aug 04 19:15:41.785075 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.784995 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5c4ce2_5922_41bb_8802_f0b554e441b4.slice/crio-8bcba879ff29da414d4e08a678bb34351db2e021024c23139424e1b70119b4d9 WatchSource:0}: Error finding container 8bcba879ff29da414d4e08a678bb34351db2e021024c23139424e1b70119b4d9: Status 404 returned error can't find the container with id 8bcba879ff29da414d4e08a678bb34351db2e021024c23139424e1b70119b4d9 Aug 04 19:15:41.786323 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.786010 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c51e61e_eef9_4ea5_8e9c_f7a8eb65a338.slice/crio-16ed2bcb5684caf8fc51b515642595a0a392e1b7f0dc361a4f0da6b071d99951 WatchSource:0}: Error finding container 16ed2bcb5684caf8fc51b515642595a0a392e1b7f0dc361a4f0da6b071d99951: Status 404 returned error can't find the container with id 16ed2bcb5684caf8fc51b515642595a0a392e1b7f0dc361a4f0da6b071d99951 Aug 04 19:15:41.786894 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.786833 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dea0b6e_1d82_46a5_b3c7_4ce3a88545c7.slice/crio-146c1a10127e4faf4444b6423e1407d6033ed9ef7c53f7167f39818aa0718316 WatchSource:0}: Error finding container 146c1a10127e4faf4444b6423e1407d6033ed9ef7c53f7167f39818aa0718316: Status 404 returned error can't find the container with id 146c1a10127e4faf4444b6423e1407d6033ed9ef7c53f7167f39818aa0718316 Aug 04 19:15:41.787915 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:41.787786 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11e729b_8b85_4b6a_a86f_3a58fb19471e.slice/crio-b30e6520fa2369238b18e9847841f5f84c9919b60a8f5651b36e1cc7ef7d89de WatchSource:0}: Error finding container b30e6520fa2369238b18e9847841f5f84c9919b60a8f5651b36e1cc7ef7d89de: Status 404 returned error can't find the container with id b30e6520fa2369238b18e9847841f5f84c9919b60a8f5651b36e1cc7ef7d89de Aug 04 19:15:41.990170 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:41.990139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:41.990298 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.990250 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:41.990344 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:41.990299 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:42.990286204 +0000 UTC m=+3.960312705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:42.030964 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:15:42.030939 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffeedbcd8e5be3b25f752d90133cc1ac.slice/crio-e41454f87590879df4239372523b9a965545b7b62d14f319198887e401ffc853 WatchSource:0}: Error finding container e41454f87590879df4239372523b9a965545b7b62d14f319198887e401ffc853: Status 404 returned error can't find the container with id e41454f87590879df4239372523b9a965545b7b62d14f319198887e401ffc853 Aug 04 19:15:42.053833 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.053812 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:42.090652 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.090623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:42.090789 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.090669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:42.090789 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090765 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:42.090900 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090828 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:43.090811703 +0000 UTC m=+4.060838211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:42.090900 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090838 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:42.090900 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090861 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:42.090900 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090882 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:42.091090 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:42.090935 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:43.090920135 +0000 UTC m=+4.060946633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:42.414584 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.414422 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:41 +0000 UTC" deadline="2028-02-28 04:01:17.262084548 +0000 UTC" Aug 04 19:15:42.414584 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.414481 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13736h45m34.847607694s" Aug 04 19:15:42.511015 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.510977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hswfg" event={"ID":"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70","Type":"ContainerStarted","Data":"565815a5ebe727c8661d5efd5503c62ae75a975bd6c9142bf818212cbbb79479"} Aug 04 19:15:42.517286 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.516930 2567 generic.go:356] "Generic (PLEG): container finished" podID="7f3042af8b8ad801d4b3bcf93179fca3" containerID="d2e4193bdf7105b94aec77caba49bb1da346a81c08798e6862d1b42c7dc7bb5c" exitCode=0 Aug 04 19:15:42.517286 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.517013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" event={"ID":"7f3042af8b8ad801d4b3bcf93179fca3","Type":"ContainerDied","Data":"d2e4193bdf7105b94aec77caba49bb1da346a81c08798e6862d1b42c7dc7bb5c"} Aug 04 19:15:42.526860 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.526833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwb2t" event={"ID":"c11e729b-8b85-4b6a-a86f-3a58fb19471e","Type":"ContainerStarted","Data":"b30e6520fa2369238b18e9847841f5f84c9919b60a8f5651b36e1cc7ef7d89de"} Aug 04 19:15:42.538535 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.538504 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerStarted","Data":"146c1a10127e4faf4444b6423e1407d6033ed9ef7c53f7167f39818aa0718316"} Aug 04 19:15:42.553602 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.553576 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" event={"ID":"b2e719c9-0d3b-471e-a451-d8c6a182a7e9","Type":"ContainerStarted","Data":"b99de7109090e4a4e7275a236d92a7bffff3cd6f3eac7f5049ac0233207517bc"} Aug 04 19:15:42.566814 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.566782 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"5a243fc1cc097e6c8a5b31be9931a8580877a463d3e65fda12e015bb342437a1"} Aug 04 19:15:42.574256 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.574225 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" event={"ID":"ffeedbcd8e5be3b25f752d90133cc1ac","Type":"ContainerStarted","Data":"e41454f87590879df4239372523b9a965545b7b62d14f319198887e401ffc853"} Aug 04 19:15:42.577852 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.577823 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tlx9" event={"ID":"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338","Type":"ContainerStarted","Data":"16ed2bcb5684caf8fc51b515642595a0a392e1b7f0dc361a4f0da6b071d99951"} Aug 04 19:15:42.590609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.590574 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k46fv" event={"ID":"0b5c4ce2-5922-41bb-8802-f0b554e441b4","Type":"ContainerStarted","Data":"8bcba879ff29da414d4e08a678bb34351db2e021024c23139424e1b70119b4d9"} Aug 04 19:15:42.605346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.605316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-679gz" event={"ID":"acc83340-7d5d-4d02-8a45-57e8242471f6","Type":"ContainerStarted","Data":"ffe3a17aa866c6a22cbe5ce87fa183586056dfc95c0dd32a7343a4f1402ca4fc"} Aug 04 19:15:42.611533 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:42.611507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" event={"ID":"ded3861d-b5e2-484f-81d7-db61175db4e4","Type":"ContainerStarted","Data":"75f998e6cf59c8f2f3fbd4e7a176ba0fe61e92872e7f2f56dc7a5143aec46f82"} Aug 04 19:15:43.000684 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.000373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:43.000843 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.000762 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:43.000843 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.000825 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:45.000807857 +0000 UTC m=+5.970834370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:43.101913 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.101875 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:43.102125 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.101977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:43.102125 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102121 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:43.102230 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102183 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:45.102164869 +0000 UTC m=+6.072191372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:43.102591 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102572 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:43.102674 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102598 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:43.102674 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102611 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:43.102674 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.102656 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:45.102641739 +0000 UTC m=+6.072668242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:43.415720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.415612 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:41 +0000 UTC" deadline="2028-03-12 23:13:44.388456738 +0000 UTC" Aug 04 19:15:43.415720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.415649 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14067h58m0.972812427s" Aug 04 19:15:43.498435 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.498404 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:43.498634 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.498532 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:43.498948 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.498921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:43.499041 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.499019 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:43.499130 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.499106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:43.499208 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:43.499186 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:43.633457 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:43.633105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" event={"ID":"7f3042af8b8ad801d4b3bcf93179fca3","Type":"ContainerStarted","Data":"e1711f37966f74350d8d233d048753fb2a8c65b705eb734276c7a30fb52160f1"} Aug 04 19:15:45.023233 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.023193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:45.023821 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.023366 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:45.023821 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.023430 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:49.023412742 +0000 UTC m=+9.993439245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:45.124223 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.124189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:45.124400 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.124276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:45.124400 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124393 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:45.124515 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124453 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:49.124434377 +0000 UTC m=+10.094460891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:45.124899 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124879 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:45.124899 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124907 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:45.125091 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124921 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:45.125091 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.124966 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:49.124951699 +0000 UTC m=+10.094978201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:45.499433 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.499364 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:45.499597 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.499485 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:45.499897 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.499878 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:45.499997 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.499978 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:45.500133 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:45.500119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:45.500223 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:45.500204 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:47.498680 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:47.498649 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:47.498680 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:47.498671 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:47.499125 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:47.498656 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:47.499125 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:47.498779 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:47.499125 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:47.498872 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:47.499125 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:47.498962 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:49.057102 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.057064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:49.057444 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.057191 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:49.057444 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.057271 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.057247718 +0000 UTC m=+18.027274230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.158311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.158363 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158477 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158500 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158517 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158530 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158561 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.158538093 +0000 UTC m=+18.128564592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:49.158580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.158584 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.158573212 +0000 UTC m=+18.128599709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.506221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.506260 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:49.506370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.506437 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.506458 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:49.506671 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:49.506593 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:51.498478 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:51.498445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:51.498478 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:51.498476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:51.498967 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:51.498445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:51.498967 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:51.498562 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:51.498967 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:51.498677 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:51.498967 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:51.498755 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:53.498744 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:53.498665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:53.499217 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:53.498665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:53.499217 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:53.498788 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:53.499217 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:53.498665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:53.499217 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:53.498850 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:53.499217 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:53.498927 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:55.498522 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:55.498486 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:55.498978 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:55.498487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:55.498978 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:55.498626 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:55.498978 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:55.498655 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:55.498978 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:55.498753 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:55.498978 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:55.498842 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:57.121616 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.121524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:57.122037 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.121676 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.122037 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.121763 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.121741552 +0000 UTC m=+34.091768070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.222834 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.222788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:57.222994 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.222882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:57.222994 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.222986 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.223133 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.222990 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:57.223133 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.223017 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:57.223133 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.223033 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:57.223133 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.223064 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.223032601 +0000 UTC m=+34.193059122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.223133 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.223098 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:13.223081607 +0000 UTC m=+34.193108124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:57.498566 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.498485 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:57.498566 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.498514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:57.498566 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:57.498485 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:57.498835 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.498600 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:57.498835 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.498666 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:15:57.498835 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:57.498762 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:59.498907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:59.498875 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:15:59.499375 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:59.498967 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:15:59.499375 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:59.498982 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:15:59.499375 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:15:59.499006 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:15:59.499375 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:59.499123 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:15:59.499375 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:15:59.499203 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:01.499093 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.498780 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:01.499780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.498780 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:01.499780 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:01.499180 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:01.499780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.498817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:01.499780 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:01.499247 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:01.499780 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:01.499335 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:01.676124 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"010f14be334d86c6bdbc9a5b55e86356dc29e00595f3c2c5891b29bf4ff0e44d"} Aug 04 19:16:01.676263 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"538d27ff479581bd19de40808423b998ce3a70ef58bc52a7cd0dbdff768d620e"} Aug 04 19:16:01.676263 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"e15baa0933d9f7a88e98b899653e60f5ffa6f779ffceca12e7e68f002253db18"} Aug 04 19:16:01.676263 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"78b7256c4cb543facbc3675f2740c2d85bb4e25e19b493511063f02922abe384"} Aug 04 19:16:01.676399 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676271 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"61481e174e0a65b8870471cf97a6b3bea3674d59bc530b208dac3fd8124feff2"} Aug 04 19:16:01.676399 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.676302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"e4aba61bdb2922940c01ab2b602ab7894e9fcb17bb166a017c999f8378012500"} Aug 04 19:16:01.677852 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.677825 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" event={"ID":"ffeedbcd8e5be3b25f752d90133cc1ac","Type":"ContainerStarted","Data":"3c72733634240c3b0e485294b32c63eee1bec55362cfe24fb994bb3219dd12dc"} Aug 04 19:16:01.679524 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.679462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tlx9" event={"ID":"6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338","Type":"ContainerStarted","Data":"8495ffde7caf323ed2a84eb8806889243e334ef8af2d8d18a6f8ecd9192e9e40"} Aug 04 19:16:01.681717 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.681686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k46fv" event={"ID":"0b5c4ce2-5922-41bb-8802-f0b554e441b4","Type":"ContainerStarted","Data":"10cf9c9b71f33290460c5738919a8ddfa54d494113d25fe0eee5441cfad81143"} Aug 04 19:16:01.683542 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.683508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" event={"ID":"ded3861d-b5e2-484f-81d7-db61175db4e4","Type":"ContainerStarted","Data":"dd1edff25b476c814960a13bf5540b64dbfa9867b364815d3ad7e05ae8ecefcd"} Aug 04 19:16:01.685371 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.685145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hswfg" event={"ID":"8749a6bd-aed8-4cd7-bd7d-51f1e2ae9c70","Type":"ContainerStarted","Data":"3d1933bc0aee6dc12231fe291ffb9eb10ab7ab716e8bf9b0587ae4831fbcc3cc"} Aug 04 19:16:01.686868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.686818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwb2t" event={"ID":"c11e729b-8b85-4b6a-a86f-3a58fb19471e","Type":"ContainerStarted","Data":"ab90d313a53beb255862d311454e36f5025982126fe3ac62c9046a58679da1fd"} Aug 04 19:16:01.688652 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.688627 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="96d7b2343a0671498824f2e0859d993a8fd62105dd0f3903caf207098a48f97d" exitCode=0 Aug 04 19:16:01.688744 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.688672 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"96d7b2343a0671498824f2e0859d993a8fd62105dd0f3903caf207098a48f97d"} Aug 04 19:16:01.690459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.690436 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" event={"ID":"b2e719c9-0d3b-471e-a451-d8c6a182a7e9","Type":"ContainerStarted","Data":"ee790e2806290dbf23287ae1aba7ecf47ce53506691b913ac11a7cc636403c65"} Aug 04 19:16:01.691735 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.691668 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-250.ec2.internal" podStartSLOduration=21.691655778 podStartE2EDuration="21.691655778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:43.651177751 +0000 UTC m=+4.621204270" watchObservedRunningTime="2026-08-04 19:16:01.691655778 +0000 UTC m=+22.661682275" Aug 04 19:16:01.712732 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.712694 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-250.ec2.internal" podStartSLOduration=21.712680409 podStartE2EDuration="21.712680409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.691133912 +0000 UTC m=+22.661160432" watchObservedRunningTime="2026-08-04 19:16:01.712680409 +0000 UTC m=+22.682706929" Aug 04 19:16:01.741354 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.741299 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k46fv" podStartSLOduration=3.685951529 podStartE2EDuration="22.741277973s" totalImagesPullingTime="19.055326444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.741031818 +0000 UTC m=+22.711058339" watchObservedRunningTime="2026-08-04 19:16:01.741277973 +0000 UTC m=+22.711304493" Aug 04 19:16:01.741723 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.741695 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kvzzm" podStartSLOduration=3.6519109849999998 podStartE2EDuration="22.741687305s" totalImagesPullingTime="19.08977632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.728029889 +0000 UTC m=+22.698056409" watchObservedRunningTime="2026-08-04 19:16:01.741687305 +0000 UTC m=+22.711713829" Aug 04 19:16:01.767938 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.767871 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pwb2t" podStartSLOduration=3.719397352 podStartE2EDuration="22.767856094s" totalImagesPullingTime="19.048458742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.754274169 +0000 UTC m=+22.724300689" watchObservedRunningTime="2026-08-04 19:16:01.767856094 +0000 UTC m=+22.737882613" Aug 04 19:16:01.768235 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.768211 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7tlx9" podStartSLOduration=7.774353476 podStartE2EDuration="22.768203791s" totalImagesPullingTime="14.993850315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.767665422 +0000 UTC m=+22.737691934" watchObservedRunningTime="2026-08-04 19:16:01.768203791 +0000 UTC m=+22.738230301" Aug 04 19:16:01.785156 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:01.785092 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hswfg" podStartSLOduration=3.655944269 podStartE2EDuration="22.785074287s" totalImagesPullingTime="19.129130018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:01.784220679 +0000 UTC m=+22.754247216" watchObservedRunningTime="2026-08-04 19:16:01.785074287 +0000 UTC m=+22.755100804" Aug 04 19:16:02.026540 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.026463 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:02.343804 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.343760 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 19:16:02.432604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.432505 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T19:16:02.343783313Z","UUID":"136e78cf-c689-40aa-a724-11da4d2fb3fb","Handler":null,"Name":"","Endpoint":""} Aug 04 19:16:02.433874 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.433851 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 19:16:02.433998 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.433885 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 19:16:02.694214 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.694120 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-679gz" event={"ID":"acc83340-7d5d-4d02-8a45-57e8242471f6","Type":"ContainerStarted","Data":"87a88cb4b2a0a7ca79dea6316fbc693cb6f2f60d9ce3e13bc8f6c7752f38cc2d"} Aug 04 19:16:02.696436 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.696408 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" event={"ID":"b2e719c9-0d3b-471e-a451-d8c6a182a7e9","Type":"ContainerStarted","Data":"88cd1eae92629a24f63f226315c727162603a7eedcd812b2bb1032b9697d25c2"} Aug 04 19:16:02.963036 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.962942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:16:02.963538 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.963515 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:16:02.977962 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:02.977909 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-679gz" podStartSLOduration=4.924644671 podStartE2EDuration="23.97788954s" totalImagesPullingTime="19.053244869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:02.708343243 +0000 UTC m=+23.678369761" watchObservedRunningTime="2026-08-04 19:16:02.97788954 +0000 UTC m=+23.947916059" Aug 04 19:16:03.498467 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.498439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:03.498606 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.498441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:03.498606 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.498558 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:03.498606 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:03.498558 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:03.498775 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:03.498650 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:03.498775 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:03.498743 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:03.635085 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.634996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:03.700384 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.700350 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" event={"ID":"b2e719c9-0d3b-471e-a451-d8c6a182a7e9","Type":"ContainerStarted","Data":"6726ba21a08911ec24ae62db57d1fc062d87b0b2ff046288bd8140e82e22a803"} Aug 04 19:16:03.733744 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:03.733719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:04.706236 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:04.706193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"7106a07e04c6596fd1132b4fdede50444f374fa9be762f66efb99be4b3b63225"} Aug 04 19:16:04.706630 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:04.706488 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 19:16:05.498820 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:05.498773 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:05.499016 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:05.498803 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:05.499016 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:05.498935 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:05.499145 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:05.499027 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:05.499145 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:05.499067 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:05.499145 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:05.499136 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:06.129251 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.129182 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:16:06.129640 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.129291 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 19:16:06.129732 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.129715 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7tlx9" Aug 04 19:16:06.145817 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.145776 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mjnmp" podStartSLOduration=5.920393788 podStartE2EDuration="27.145764053s" totalImagesPullingTime="21.225370265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:03.729736139 +0000 UTC m=+24.699762659" watchObservedRunningTime="2026-08-04 19:16:06.145764053 +0000 UTC m=+27.115790572" Aug 04 19:16:06.189175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.189029 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:06.715472 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.715444 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="cbf8e5d9c963ebf6b060b20a7fc51d6cd9e6a8ec5451cc920a166bd959fbf407" exitCode=0 Aug 04 19:16:06.715593 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.715528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"cbf8e5d9c963ebf6b060b20a7fc51d6cd9e6a8ec5451cc920a166bd959fbf407"} Aug 04 19:16:06.719220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.719192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" event={"ID":"ba41eb7a-3667-4dee-b319-281fbafe2993","Type":"ContainerStarted","Data":"2a1816f84e18223b105bd3f31b411e22443a76465b40115aa762d6baca29c0ec"} Aug 04 19:16:06.719468 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.719453 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:06.719541 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.719534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:06.733798 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.733776 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:06.761886 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:06.761820 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" podStartSLOduration=8.638121565 podStartE2EDuration="27.761806916s" totalImagesPullingTime="19.123685351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.760496082 +0000 UTC m=+27.730522592" watchObservedRunningTime="2026-08-04 19:16:06.761806916 +0000 UTC m=+27.731833434" Aug 04 19:16:07.498975 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:07.498941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:07.499866 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:07.499095 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:07.499866 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:07.498954 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:07.499866 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:07.498941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:07.499866 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:07.499176 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:07.499866 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:07.499277 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:07.722238 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:07.722175 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:07.737597 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:07.737570 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:08.525105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.524773 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ldng"] Aug 04 19:16:08.525105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.525019 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:08.525596 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:08.525141 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:08.527408 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.527385 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4hwg"] Aug 04 19:16:08.527521 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.527459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:08.527580 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:08.527538 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:08.537285 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.537265 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62hbz"] Aug 04 19:16:08.537401 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.537334 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:08.537442 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:08.537400 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:08.618578 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.618487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:08.707909 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.707876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:08.725324 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.725298 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="03af73dd0975fd3eaa5eb625b7ac57ebcd506a09dbdde96532e2bb217cee0655" exitCode=0 Aug 04 19:16:08.725498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.725392 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"03af73dd0975fd3eaa5eb625b7ac57ebcd506a09dbdde96532e2bb217cee0655"} Aug 04 19:16:08.808461 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:08.808429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:09.673347 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:09.673315 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:09.773483 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:09.773430 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:10.019089 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.019039 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:10.074026 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.073992 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:10.498160 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.498133 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:10.498282 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.498164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:10.498282 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.498133 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:10.498282 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:10.498234 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:10.498405 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:10.498317 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:10.498405 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:10.498384 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:10.527604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.527571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:10.731905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.731821 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="d40dd2551a1dfd34fc246fd3db8e2c2b77d1874949a0953833946290313d41be" exitCode=0 Aug 04 19:16:10.731905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.731871 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"d40dd2551a1dfd34fc246fd3db8e2c2b77d1874949a0953833946290313d41be"} Aug 04 19:16:10.905166 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:10.905131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:12.129509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.129473 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:12.233986 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.233954 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:12.339353 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.339317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:12.446603 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.446515 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:12.498392 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.498360 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:12.498392 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.498392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:12.498618 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:12.498395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:12.498618 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:12.498489 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:12.498618 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:12.498562 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:12.498756 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:12.498671 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:13.146987 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.146953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:13.147441 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.147123 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:13.147441 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.147183 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret podName:524abca7-3ea5-44a3-a90e-fa2c5b9e6c96 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:45.147167153 +0000 UTC m=+66.117193650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret") pod "global-pull-secret-syncer-b4hwg" (UID: "524abca7-3ea5-44a3-a90e-fa2c5b9e6c96") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:16:13.247503 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.247473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:13.247659 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.247509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:13.247659 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247614 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:13.247659 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247626 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:16:13.247659 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247650 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:16:13.247659 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247662 2567 projected.go:196] Error preparing data for projected volume kube-api-access-s2knx for pod openshift-network-diagnostics/network-check-target-62hbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:13.247827 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247686 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs podName:d2912b55-9498-4cc6-ab05-fb8475e1deb2 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:45.247668068 +0000 UTC m=+66.217694585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs") pod "network-metrics-daemon-2ldng" (UID: "d2912b55-9498-4cc6-ab05-fb8475e1deb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:16:13.247827 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:13.247703 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx podName:6202f887-72c4-4ca7-815e-6484c7342e01 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:45.247693875 +0000 UTC m=+66.217720379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2knx" (UniqueName: "kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx") pod "network-check-target-62hbz" (UID: "6202f887-72c4-4ca7-815e-6484c7342e01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:16:13.391151 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.391115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:13.591917 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.591888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:13.953549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:13.953473 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:14.064005 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.063950 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:14.185481 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.185445 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:14.498773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.498745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:14.498773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.498783 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:14.499013 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:14.498880 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:14.499013 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:14.498917 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:14.499013 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.498961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:14.499205 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:14.499068 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:14.592860 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:14.592815 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:15.192315 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:15.192275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:15.738450 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:15.738420 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:15.850658 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:15.850411 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:15.988121 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:15.988030 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:16.196753 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.196720 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:16.498169 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.498145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:16.498341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.498228 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:16.498341 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:16.498231 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:16.498341 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:16.498308 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:16.498341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.498234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:16.498483 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:16.498391 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:16.621550 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.621516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:16.748072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.748026 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="6f7bef251a983eb77b91799a7ba6df99b42f864d4a7d36db52c6577603d73657" exitCode=0 Aug 04 19:16:16.748273 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:16.748081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"6f7bef251a983eb77b91799a7ba6df99b42f864d4a7d36db52c6577603d73657"} Aug 04 19:16:17.753084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:17.753041 2567 generic.go:356] "Generic (PLEG): container finished" podID="5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7" containerID="adfdafe79e659b32c2a5983635f7dfc87a6f3c93f1c301de280b5eb16969bd9c" exitCode=0 Aug 04 19:16:17.753451 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:17.753106 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerDied","Data":"adfdafe79e659b32c2a5983635f7dfc87a6f3c93f1c301de280b5eb16969bd9c"} Aug 04 19:16:18.499023 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:18.498989 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:18.499237 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:18.499112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:18.499237 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:18.499136 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:18.499237 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:18.499126 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:18.499237 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:18.499198 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:18.499451 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:18.499284 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:18.758749 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:18.758671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5k47" event={"ID":"5dea0b6e-1d82-46a5-b3c7-4ce3a88545c7","Type":"ContainerStarted","Data":"301b0ba0e7db5bc75b3f675ebbfa698c29d3e8511208c2b0b166065057a30599"} Aug 04 19:16:18.782747 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:18.782700 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q5k47" podStartSLOduration=8.762615348 podStartE2EDuration="39.782687383s" totalImagesPullingTime="31.020072035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:18.78113564 +0000 UTC m=+39.751162158" watchObservedRunningTime="2026-08-04 19:16:18.782687383 +0000 UTC m=+39.752713901" Aug 04 19:16:20.498910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:20.498874 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:20.499305 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:20.498879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:20.499305 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:20.498978 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:20.499305 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:20.498995 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:20.499305 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:20.499126 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:20.499305 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:20.499184 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:22.498302 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:22.498265 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:22.498302 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:22.498309 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:22.498712 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:22.498390 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:22.498712 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:22.498431 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:22.498712 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:22.498470 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:22.498712 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:22.498552 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:24.498651 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:24.498621 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:24.499123 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:24.498718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:24.499123 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:24.498731 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:24.499123 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:24.498792 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:24.499123 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:24.498801 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:24.499123 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:24.498882 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:26.498608 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:26.498564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:26.499117 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:26.498570 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:26.499117 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:26.498698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:26.499117 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:26.498692 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:26.499117 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:26.498794 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:26.499117 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:26.498864 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:28.498850 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:28.498812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:28.498850 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:28.498834 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:28.499399 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:28.498812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:28.499399 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:28.498950 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:28.499399 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:28.498985 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:28.499399 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:28.499032 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:30.498987 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.498959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:30.499295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.498958 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:30.499295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.499080 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:30.499295 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:30.499072 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ldng" podUID="d2912b55-9498-4cc6-ab05-fb8475e1deb2" Aug 04 19:16:30.499295 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:30.499165 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4hwg" podUID="524abca7-3ea5-44a3-a90e-fa2c5b9e6c96" Aug 04 19:16:30.499295 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:30.499223 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62hbz" podUID="6202f887-72c4-4ca7-815e-6484c7342e01" Aug 04 19:16:30.883755 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.883733 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-250.ec2.internal" event="NodeReady" Aug 04 19:16:30.883878 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.883814 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 19:16:30.926024 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.925998 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ffd8f474-bcmbw"] Aug 04 19:16:30.943723 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.943705 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5c8mm"] Aug 04 19:16:30.943849 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.943832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.946620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.946604 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 19:16:30.946729 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.946646 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 19:16:30.946794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.946726 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5pc5\"" Aug 04 19:16:30.947778 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.947761 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 19:16:30.953276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.953253 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 19:16:30.957542 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.957520 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fcklt"] Aug 04 19:16:30.957658 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.957641 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:30.960023 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.959997 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 19:16:30.960147 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.960095 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svf4z\"" Aug 04 19:16:30.960204 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.960189 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 19:16:30.973811 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.973793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ffd8f474-bcmbw"] Aug 04 19:16:30.973893 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.973842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:30.973893 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.973843 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c8mm"] Aug 04 19:16:30.973972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.973897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fcklt"] Aug 04 19:16:30.976872 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.976858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 19:16:30.976962 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.976876 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 19:16:30.977449 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.977429 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 19:16:30.977667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.977654 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sds9s\"" Aug 04 19:16:30.982756 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.982739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-tls\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.982844 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.982766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-bound-sa-token\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.982960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.982791 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3db70ed7-f73a-47ca-a650-a124d82fb08c-metrics-tls\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983157 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxfh\" (UniqueName: \"kubernetes.io/projected/3db70ed7-f73a-47ca-a650-a124d82fb08c-kube-api-access-dmxfh\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rvr\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-kube-api-access-54rvr\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983419 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-certificates\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-trusted-ca\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-installation-pull-secrets\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-image-registry-private-configuration\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983693 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b55cc7a-4785-4725-900f-97ac50a1658e-ca-trust-extracted\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3db70ed7-f73a-47ca-a650-a124d82fb08c-config-volume\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:30.985997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:30.983868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db70ed7-f73a-47ca-a650-a124d82fb08c-tmp-dir\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.085264 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-tls\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085264 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085263 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-cert\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.085424 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-bound-sa-token\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085424 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3db70ed7-f73a-47ca-a650-a124d82fb08c-metrics-tls\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.085424 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxfh\" (UniqueName: \"kubernetes.io/projected/3db70ed7-f73a-47ca-a650-a124d82fb08c-kube-api-access-dmxfh\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.085424 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rvr\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-kube-api-access-54rvr\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085612 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmztw\" (UniqueName: \"kubernetes.io/projected/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-kube-api-access-cmztw\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.085612 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-certificates\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085612 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-trusted-ca\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085612 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-installation-pull-secrets\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-image-registry-private-configuration\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b55cc7a-4785-4725-900f-97ac50a1658e-ca-trust-extracted\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.085808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.085663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3db70ed7-f73a-47ca-a650-a124d82fb08c-config-volume\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.086191 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db70ed7-f73a-47ca-a650-a124d82fb08c-tmp-dir\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.086358 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b55cc7a-4785-4725-900f-97ac50a1658e-ca-trust-extracted\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.086682 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db70ed7-f73a-47ca-a650-a124d82fb08c-tmp-dir\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.086682 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086473 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-certificates\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.086682 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086587 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3db70ed7-f73a-47ca-a650-a124d82fb08c-config-volume\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.086863 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.086848 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b55cc7a-4785-4725-900f-97ac50a1658e-trusted-ca\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.089891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.089869 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3db70ed7-f73a-47ca-a650-a124d82fb08c-metrics-tls\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.089953 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.089928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-image-registry-private-configuration\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.090163 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.090142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b55cc7a-4785-4725-900f-97ac50a1658e-installation-pull-secrets\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.090318 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.090303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-registry-tls\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.092960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.092937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-bound-sa-token\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.093570 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.093556 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxfh\" (UniqueName: \"kubernetes.io/projected/3db70ed7-f73a-47ca-a650-a124d82fb08c-kube-api-access-dmxfh\") pod \"dns-default-5c8mm\" (UID: \"3db70ed7-f73a-47ca-a650-a124d82fb08c\") " pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.093768 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.093752 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rvr\" (UniqueName: \"kubernetes.io/projected/7b55cc7a-4785-4725-900f-97ac50a1658e-kube-api-access-54rvr\") pod \"image-registry-ffd8f474-bcmbw\" (UID: \"7b55cc7a-4785-4725-900f-97ac50a1658e\") " pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.187360 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.187305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-cert\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.187360 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.187341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmztw\" (UniqueName: \"kubernetes.io/projected/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-kube-api-access-cmztw\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.189159 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.189136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-cert\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.194732 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.194715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmztw\" (UniqueName: \"kubernetes.io/projected/cc5463a9-37d5-45ae-84f1-dc3858b8c6c9-kube-api-access-cmztw\") pod \"ingress-canary-fcklt\" (UID: \"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9\") " pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.255229 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.255201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:31.266405 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.266389 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:31.281471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.281441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fcklt" Aug 04 19:16:31.708549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.708476 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ffd8f474-bcmbw"] Aug 04 19:16:31.709390 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.709303 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c8mm"] Aug 04 19:16:31.710907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.710830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fcklt"] Aug 04 19:16:31.715381 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:31.715362 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db70ed7_f73a_47ca_a650_a124d82fb08c.slice/crio-50d43e6e9850d7de81597c77b2898cf8888e72fa21cd570bd1660e0b5b7338d7 WatchSource:0}: Error finding container 50d43e6e9850d7de81597c77b2898cf8888e72fa21cd570bd1660e0b5b7338d7: Status 404 returned error can't find the container with id 50d43e6e9850d7de81597c77b2898cf8888e72fa21cd570bd1660e0b5b7338d7 Aug 04 19:16:31.715608 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:31.715584 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b55cc7a_4785_4725_900f_97ac50a1658e.slice/crio-949fe2f8982829a292a5abd840bff6804392c28fe90029428349f840bd745bb4 WatchSource:0}: Error finding container 949fe2f8982829a292a5abd840bff6804392c28fe90029428349f840bd745bb4: Status 404 returned error can't find the container with id 949fe2f8982829a292a5abd840bff6804392c28fe90029428349f840bd745bb4 Aug 04 19:16:31.715922 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:31.715875 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5463a9_37d5_45ae_84f1_dc3858b8c6c9.slice/crio-bc2ec1a9d3ae56e9930125fd4724397634251baff72555858ceaed0be39b945b WatchSource:0}: Error finding container bc2ec1a9d3ae56e9930125fd4724397634251baff72555858ceaed0be39b945b: Status 404 returned error can't find the container with id bc2ec1a9d3ae56e9930125fd4724397634251baff72555858ceaed0be39b945b Aug 04 19:16:31.785059 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.785029 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fcklt" event={"ID":"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9","Type":"ContainerStarted","Data":"bc2ec1a9d3ae56e9930125fd4724397634251baff72555858ceaed0be39b945b"} Aug 04 19:16:31.785989 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.785968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c8mm" event={"ID":"3db70ed7-f73a-47ca-a650-a124d82fb08c","Type":"ContainerStarted","Data":"50d43e6e9850d7de81597c77b2898cf8888e72fa21cd570bd1660e0b5b7338d7"} Aug 04 19:16:31.786904 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:31.786885 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" event={"ID":"7b55cc7a-4785-4725-900f-97ac50a1658e","Type":"ContainerStarted","Data":"949fe2f8982829a292a5abd840bff6804392c28fe90029428349f840bd745bb4"} Aug 04 19:16:32.498996 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.498879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:32.498996 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.498902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:32.499326 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.498891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503091 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503116 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503128 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503177 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503177 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:16:32.503414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.503234 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bhjx\"" Aug 04 19:16:32.792602 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.792567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" event={"ID":"7b55cc7a-4785-4725-900f-97ac50a1658e","Type":"ContainerStarted","Data":"621f9f410f067069c9f10bcb59ecb6ad83fd2aba03da1c5ede8c6f26a81df4b8"} Aug 04 19:16:32.793008 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.792740 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:32.812472 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:32.812425 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" podStartSLOduration=2.812408883 podStartE2EDuration="2.812408883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:32.81136374 +0000 UTC m=+53.781390279" watchObservedRunningTime="2026-08-04 19:16:32.812408883 +0000 UTC m=+53.782435401" Aug 04 19:16:33.520728 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.520697 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:16:33.534925 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.534865 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.535922 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.535899 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:16:33.538142 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538084 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 19:16:33.538253 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538152 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 19:16:33.538253 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538199 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 19:16:33.538342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538152 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5wcwc\"" Aug 04 19:16:33.538342 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538306 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 19:16:33.538566 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538521 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 19:16:33.538779 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.538761 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 19:16:33.539035 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.539021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 19:16:33.612384 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.612532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.612532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w88p9\" (UniqueName: \"kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.612532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612511 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.612532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.612680 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.612547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713557 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713557 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713730 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713577 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713730 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.713868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.713790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w88p9\" (UniqueName: \"kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.714283 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.714265 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.714381 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.714311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.714381 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.714366 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.715954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.715929 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.716361 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.716338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.722631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.722608 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w88p9\" (UniqueName: \"kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9\") pod \"console-585fc4cc99-4s4xr\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:33.847295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:33.847273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:34.569488 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.569460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:16:34.572600 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:34.572519 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0a97f2_586c_4b34_8e1a_975cf45fe36b.slice/crio-6d599a825641aa65bf3c4263dde82c488d99d9252ea73e912568715270831acf WatchSource:0}: Error finding container 6d599a825641aa65bf3c4263dde82c488d99d9252ea73e912568715270831acf: Status 404 returned error can't find the container with id 6d599a825641aa65bf3c4263dde82c488d99d9252ea73e912568715270831acf Aug 04 19:16:34.799040 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.799007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fcklt" event={"ID":"cc5463a9-37d5-45ae-84f1-dc3858b8c6c9","Type":"ContainerStarted","Data":"bf55c73d6f9fc22359d51fe00d5e6d6cc1987f20d7f9323dd0898ddf3f8f8e88"} Aug 04 19:16:34.800763 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.800739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c8mm" event={"ID":"3db70ed7-f73a-47ca-a650-a124d82fb08c","Type":"ContainerStarted","Data":"551a0ea21b2a5db65d92daffe055e650ee2501a3053b1ed2145e2ecce5ffd8b5"} Aug 04 19:16:34.800871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.800781 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c8mm" event={"ID":"3db70ed7-f73a-47ca-a650-a124d82fb08c","Type":"ContainerStarted","Data":"19ed11c8fccae1d8e5ef68ae53fb324d68329b904a52f3b04f04cb854f8f3264"} Aug 04 19:16:34.800871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.800803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:34.801931 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.801905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fc4cc99-4s4xr" event={"ID":"9b0a97f2-586c-4b34-8e1a-975cf45fe36b","Type":"ContainerStarted","Data":"6d599a825641aa65bf3c4263dde82c488d99d9252ea73e912568715270831acf"} Aug 04 19:16:34.813816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.813768 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fcklt" podStartSLOduration=2.089543106 podStartE2EDuration="4.813751834s" totalImagesPullingTime="2.724208728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:34.813488434 +0000 UTC m=+55.783514953" watchObservedRunningTime="2026-08-04 19:16:34.813751834 +0000 UTC m=+55.783778333" Aug 04 19:16:34.829706 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:34.829667 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5c8mm" podStartSLOduration=2.110616376 podStartE2EDuration="4.829655965s" totalImagesPullingTime="2.719039589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:34.828936624 +0000 UTC m=+55.798963144" watchObservedRunningTime="2026-08-04 19:16:34.829655965 +0000 UTC m=+55.799682483" Aug 04 19:16:39.747328 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:39.747300 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfpxw" Aug 04 19:16:39.815268 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:39.815240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fc4cc99-4s4xr" event={"ID":"9b0a97f2-586c-4b34-8e1a-975cf45fe36b","Type":"ContainerStarted","Data":"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21"} Aug 04 19:16:41.274626 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.274577 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-585fc4cc99-4s4xr" podStartSLOduration=3.919837919 podStartE2EDuration="8.27455966s" totalImagesPullingTime="4.354721741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:39.832542174 +0000 UTC m=+60.802568693" watchObservedRunningTime="2026-08-04 19:16:41.27455966 +0000 UTC m=+62.244586178" Aug 04 19:16:41.275472 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.275450 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6pvgm"] Aug 04 19:16:41.305466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.305439 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pvgm"] Aug 04 19:16:41.305569 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.305561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.309193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.309171 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 19:16:41.309288 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.309176 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 19:16:41.309337 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.309291 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6xxp\"" Aug 04 19:16:41.309498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.309484 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 19:16:41.310810 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.310792 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 19:16:41.370941 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.370913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc129f0a-75ee-4332-a393-60520f6c4ac7-crio-socket\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.371044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.370961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc129f0a-75ee-4332-a393-60520f6c4ac7-data-volume\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.371044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.370981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58xn\" (UniqueName: \"kubernetes.io/projected/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-api-access-l58xn\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.371044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.371029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.371175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.371112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471404 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471377 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc129f0a-75ee-4332-a393-60520f6c4ac7-crio-socket\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471499 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471420 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc129f0a-75ee-4332-a393-60520f6c4ac7-data-volume\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471499 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58xn\" (UniqueName: \"kubernetes.io/projected/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-api-access-l58xn\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471499 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471499 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc129f0a-75ee-4332-a393-60520f6c4ac7-crio-socket\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471648 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.471648 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.471586 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.471648 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.471599 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.471743 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.471655 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.971629632 +0000 UTC m=+62.941656138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.471743 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.471672 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.971663364 +0000 UTC m=+62.941689865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : configmap "kube-rbac-proxy" not found Aug 04 19:16:41.471743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.471689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc129f0a-75ee-4332-a393-60520f6c4ac7-data-volume\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.487333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.487306 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58xn\" (UniqueName: \"kubernetes.io/projected/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-api-access-l58xn\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.976344 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.976317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.976497 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:41.976359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:41.976497 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.976449 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.976497 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.976493 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.976479581 +0000 UTC m=+63.946506078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : configmap "kube-rbac-proxy" not found Aug 04 19:16:41.976624 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.976449 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.976624 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:41.976577 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.976564203 +0000 UTC m=+63.946590700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.983856 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:42.983820 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:42.984221 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:42.983868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:42.984221 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:42.983979 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:42.984221 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:42.984037 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.98402295 +0000 UTC m=+65.954049446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : configmap "kube-rbac-proxy" not found Aug 04 19:16:42.984221 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:42.983978 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.984221 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:42.984158 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.984142273 +0000 UTC m=+65.954168787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:43.653246 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:43.653216 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-585fc4cc99-4s4xr_9b0a97f2-586c-4b34-8e1a-975cf45fe36b/console/0.log" Aug 04 19:16:43.847471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:43.847447 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:43.847673 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:43.847497 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:16:43.848709 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:43.848688 2567 patch_prober.go:28] interesting pod/console-585fc4cc99-4s4xr container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body= Aug 04 19:16:43.848766 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:43.848728 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-585fc4cc99-4s4xr" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" Aug 04 19:16:44.252510 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:44.252486 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c8mm_3db70ed7-f73a-47ca-a650-a124d82fb08c/dns/0.log" Aug 04 19:16:44.452386 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:44.452366 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c8mm_3db70ed7-f73a-47ca-a650-a124d82fb08c/kube-rbac-proxy/0.log" Aug 04 19:16:44.807463 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:44.807440 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5c8mm" Aug 04 19:16:44.998831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:44.998794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:44.998831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:44.998839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:44.999072 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:44.998912 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:44.999072 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:44.998946 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:44.999072 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:44.998961 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:48.998948413 +0000 UTC m=+69.968974910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : configmap "kube-rbac-proxy" not found Aug 04 19:16:44.999072 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:44.999005 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:48.998989199 +0000 UTC m=+69.969015710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:45.199885 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.199821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:45.202587 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.202568 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:16:45.212829 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.212805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/524abca7-3ea5-44a3-a90e-fa2c5b9e6c96-original-pull-secret\") pod \"global-pull-secret-syncer-b4hwg\" (UID: \"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96\") " pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:45.300248 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.300221 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:45.300552 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.300287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:45.302973 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.302951 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 19:16:45.303045 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.303012 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:16:45.313067 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.313027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2912b55-9498-4cc6-ab05-fb8475e1deb2-metrics-certs\") pod \"network-metrics-daemon-2ldng\" (UID: \"d2912b55-9498-4cc6-ab05-fb8475e1deb2\") " pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:45.314018 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.314000 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 19:16:45.323087 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.323044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2knx\" (UniqueName: \"kubernetes.io/projected/6202f887-72c4-4ca7-815e-6484c7342e01-kube-api-access-s2knx\") pod \"network-check-target-62hbz\" (UID: \"6202f887-72c4-4ca7-815e-6484c7342e01\") " pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:45.415868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.415847 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:16:45.424388 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.424371 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ldng" Aug 04 19:16:45.424664 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.424647 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bhjx\"" Aug 04 19:16:45.428088 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.428069 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4hwg" Aug 04 19:16:45.431496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.431478 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:45.571339 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.571310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ldng"] Aug 04 19:16:45.574543 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:45.574518 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2912b55_9498_4cc6_ab05_fb8475e1deb2.slice/crio-e089f04390d6b04f7768a3014410e800c55c1e13ab79aabb19dae4389f5df191 WatchSource:0}: Error finding container e089f04390d6b04f7768a3014410e800c55c1e13ab79aabb19dae4389f5df191: Status 404 returned error can't find the container with id e089f04390d6b04f7768a3014410e800c55c1e13ab79aabb19dae4389f5df191 Aug 04 19:16:45.804317 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.804293 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4hwg"] Aug 04 19:16:45.806156 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:45.805965 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524abca7_3ea5_44a3_a90e_fa2c5b9e6c96.slice/crio-a110724743470fea7df8b38d59caea7bd8cc3e94b17a5707df87ef72dcb9ee62 WatchSource:0}: Error finding container a110724743470fea7df8b38d59caea7bd8cc3e94b17a5707df87ef72dcb9ee62: Status 404 returned error can't find the container with id a110724743470fea7df8b38d59caea7bd8cc3e94b17a5707df87ef72dcb9ee62 Aug 04 19:16:45.807143 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.807120 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62hbz"] Aug 04 19:16:45.809827 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:45.809805 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6202f887_72c4_4ca7_815e_6484c7342e01.slice/crio-7dccf40180e45b5409c44e652d0f483ace4ef35fbb48a2ea4ec075d13621ca25 WatchSource:0}: Error finding container 7dccf40180e45b5409c44e652d0f483ace4ef35fbb48a2ea4ec075d13621ca25: Status 404 returned error can't find the container with id 7dccf40180e45b5409c44e652d0f483ace4ef35fbb48a2ea4ec075d13621ca25 Aug 04 19:16:45.828247 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.828222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ldng" event={"ID":"d2912b55-9498-4cc6-ab05-fb8475e1deb2","Type":"ContainerStarted","Data":"e089f04390d6b04f7768a3014410e800c55c1e13ab79aabb19dae4389f5df191"} Aug 04 19:16:45.829108 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.829084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b4hwg" event={"ID":"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96","Type":"ContainerStarted","Data":"a110724743470fea7df8b38d59caea7bd8cc3e94b17a5707df87ef72dcb9ee62"} Aug 04 19:16:45.830109 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.830090 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62hbz" event={"ID":"6202f887-72c4-4ca7-815e-6484c7342e01","Type":"ContainerStarted","Data":"7dccf40180e45b5409c44e652d0f483ace4ef35fbb48a2ea4ec075d13621ca25"} Aug 04 19:16:45.854448 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:45.854433 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwb2t_c11e729b-8b85-4b6a-a86f-3a58fb19471e/dns-node-resolver/0.log" Aug 04 19:16:46.254273 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:46.254189 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-ffd8f474-bcmbw_7b55cc7a-4785-4725-900f-97ac50a1658e/registry/0.log" Aug 04 19:16:46.835944 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:46.835909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ldng" event={"ID":"d2912b55-9498-4cc6-ab05-fb8475e1deb2","Type":"ContainerStarted","Data":"4810124aecfbad5ae4008362d539417a1f8fe047bf8e7466421f8242aef0221f"} Aug 04 19:16:46.854718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:46.854695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k46fv_0b5c4ce2-5922-41bb-8802-f0b554e441b4/node-ca/0.log" Aug 04 19:16:47.452516 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:47.452486 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fcklt_cc5463a9-37d5-45ae-84f1-dc3858b8c6c9/serve-healthcheck-canary/0.log" Aug 04 19:16:47.842629 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:47.842582 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ldng" event={"ID":"d2912b55-9498-4cc6-ab05-fb8475e1deb2","Type":"ContainerStarted","Data":"48844fa435fd1cad432f58bcdf8e6fe887636d13494618d80af087cfd88a3a56"} Aug 04 19:16:47.859976 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:47.859930 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2ldng" podStartSLOduration=67.774524876 podStartE2EDuration="1m8.859902741s" totalImagesPullingTime="1.085377865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:47.858429424 +0000 UTC m=+68.828455943" watchObservedRunningTime="2026-08-04 19:16:47.859902741 +0000 UTC m=+68.829929260" Aug 04 19:16:48.979799 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:48.979765 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:49.029021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:49.029088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:49.029200 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:49.029266 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:57.029249046 +0000 UTC m=+77.999275558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : configmap "kube-rbac-proxy" not found Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:49.029622 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:49.029696 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:49.029668 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls podName:bc129f0a-75ee-4332-a393-60520f6c4ac7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:57.029653927 +0000 UTC m=+77.999680437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pvgm" (UID: "bc129f0a-75ee-4332-a393-60520f6c4ac7") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:49.852430 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:49.852393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62hbz" event={"ID":"6202f887-72c4-4ca7-815e-6484c7342e01","Type":"ContainerStarted","Data":"77afa7b3c92774ae9b46562bc38ccda074b2ec62f32398f1a08878a23f5d02f9"} Aug 04 19:16:49.852617 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:49.852522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:16:49.871125 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:49.871085 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-62hbz" podStartSLOduration=67.604888598 podStartE2EDuration="1m10.871066507s" totalImagesPullingTime="3.266177909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:49.870542841 +0000 UTC m=+70.840569361" watchObservedRunningTime="2026-08-04 19:16:49.871066507 +0000 UTC m=+70.841093021" Aug 04 19:16:50.857973 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:50.857932 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b4hwg" event={"ID":"524abca7-3ea5-44a3-a90e-fa2c5b9e6c96","Type":"ContainerStarted","Data":"4ee68a4c4f302e5991b87dd5b4e5fc682ed8323038f784c1c98dd8ec2db4355a"} Aug 04 19:16:50.874602 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:50.874560 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b4hwg" podStartSLOduration=67.051515623 podStartE2EDuration="1m11.874546517s" totalImagesPullingTime="4.823030894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:50.872917502 +0000 UTC m=+71.842944018" watchObservedRunningTime="2026-08-04 19:16:50.874546517 +0000 UTC m=+71.844573035" Aug 04 19:16:51.261091 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.260875 2567 patch_prober.go:28] interesting pod/image-registry-ffd8f474-bcmbw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 19:16:51.261091 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.260928 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" podUID="7b55cc7a-4785-4725-900f-97ac50a1658e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:16:51.696842 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.696815 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn"] Aug 04 19:16:51.700644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.700492 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.703708 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.703686 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 04 19:16:51.703814 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.703737 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 04 19:16:51.703814 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.703787 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 19:16:51.704890 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.704869 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 19:16:51.704988 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.704869 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 19:16:51.704988 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.704932 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kgt7f\"" Aug 04 19:16:51.704988 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.704945 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 04 19:16:51.710965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.710946 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pkppb"] Aug 04 19:16:51.713138 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.713117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn"] Aug 04 19:16:51.713259 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.713243 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.716227 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.716209 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 19:16:51.716324 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.716214 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 19:16:51.716324 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.716254 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 19:16:51.716686 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.716670 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mrvps\"" Aug 04 19:16:51.851471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851442 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851510 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5a17e53-02ff-463a-9919-160910ac2e10-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tprb\" (UniqueName: \"kubernetes.io/projected/a5a17e53-02ff-463a-9919-160910ac2e10-kube-api-access-8tprb\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-textfile\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-root\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-wtmp\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-metrics-client-ca\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcln2\" (UniqueName: \"kubernetes.io/projected/08a85812-f966-4d91-bfe4-39a86252ea9a-kube-api-access-tcln2\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.851859 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-sys\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.852069 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.851865 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.952549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcln2\" (UniqueName: \"kubernetes.io/projected/08a85812-f966-4d91-bfe4-39a86252ea9a-kube-api-access-tcln2\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.952549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.952549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-sys\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-sys\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5a17e53-02ff-463a-9919-160910ac2e10-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:51.952871 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:51.952899 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.952898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tprb\" (UniqueName: \"kubernetes.io/projected/a5a17e53-02ff-463a-9919-160910ac2e10-kube-api-access-8tprb\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953011 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:51.952945 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls podName:a5a17e53-02ff-463a-9919-160910ac2e10 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:52.452925141 +0000 UTC m=+73.422951648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-q4gdn" (UID: "a5a17e53-02ff-463a-9919-160910ac2e10") : secret "kube-state-metrics-tls" not found Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:51.953093 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls podName:08a85812-f966-4d91-bfe4-39a86252ea9a nodeName:}" failed. No retries permitted until 2026-08-04 19:16:52.453078514 +0000 UTC m=+73.423105016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls") pod "node-exporter-pkppb" (UID: "08a85812-f966-4d91-bfe4-39a86252ea9a") : secret "node-exporter-tls" not found Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-textfile\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-root\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-wtmp\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-metrics-client-ca\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5a17e53-02ff-463a-9919-160910ac2e10-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-textfile\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-wtmp\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08a85812-f966-4d91-bfe4-39a86252ea9a-root\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.953957 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953957 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5a17e53-02ff-463a-9919-160910ac2e10-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.953957 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.953710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a85812-f966-4d91-bfe4-39a86252ea9a-metrics-client-ca\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.955089 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.955071 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:51.955604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.955586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.962274 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.962250 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tprb\" (UniqueName: \"kubernetes.io/projected/a5a17e53-02ff-463a-9919-160910ac2e10-kube-api-access-8tprb\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:51.963007 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:51.962981 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcln2\" (UniqueName: \"kubernetes.io/projected/08a85812-f966-4d91-bfe4-39a86252ea9a-kube-api-access-tcln2\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:52.456790 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.456753 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:52.456982 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.456802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:52.459109 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.459085 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a17e53-02ff-463a-9919-160910ac2e10-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-q4gdn\" (UID: \"a5a17e53-02ff-463a-9919-160910ac2e10\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:52.459209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.459121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08a85812-f966-4d91-bfe4-39a86252ea9a-node-exporter-tls\") pod \"node-exporter-pkppb\" (UID: \"08a85812-f966-4d91-bfe4-39a86252ea9a\") " pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:52.609694 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.609664 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" Aug 04 19:16:52.621460 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.621432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkppb" Aug 04 19:16:52.629519 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:52.629493 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a85812_f966_4d91_bfe4_39a86252ea9a.slice/crio-2fc4c175b6037e903cfee54e7b81ab6e2fdfbf75b3de5bb9fb621e09ab81ad91 WatchSource:0}: Error finding container 2fc4c175b6037e903cfee54e7b81ab6e2fdfbf75b3de5bb9fb621e09ab81ad91: Status 404 returned error can't find the container with id 2fc4c175b6037e903cfee54e7b81ab6e2fdfbf75b3de5bb9fb621e09ab81ad91 Aug 04 19:16:52.735677 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.735596 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn"] Aug 04 19:16:52.738680 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:52.738654 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a17e53_02ff_463a_9919_160910ac2e10.slice/crio-a3726956b806b96c872032c484c456b9d8c6105300420beb8bccabb9f1712c83 WatchSource:0}: Error finding container a3726956b806b96c872032c484c456b9d8c6105300420beb8bccabb9f1712c83: Status 404 returned error can't find the container with id a3726956b806b96c872032c484c456b9d8c6105300420beb8bccabb9f1712c83 Aug 04 19:16:52.782771 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.782749 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 19:16:52.785776 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.785762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.788837 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.788818 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 19:16:52.788932 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.788818 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 19:16:52.789114 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789100 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 19:16:52.789215 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789123 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 19:16:52.789215 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789145 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 19:16:52.789215 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 19:16:52.789215 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789211 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2b24j\"" Aug 04 19:16:52.789426 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 19:16:52.789426 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 19:16:52.789426 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.789217 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 19:16:52.804215 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.804198 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 19:16:52.859574 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859605 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859792 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859792 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859792 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859880 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tmz\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-kube-api-access-w8tmz\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859880 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-web-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859880 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859905 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-out\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.859970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.859946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.864886 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.864863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkppb" event={"ID":"08a85812-f966-4d91-bfe4-39a86252ea9a","Type":"ContainerStarted","Data":"2fc4c175b6037e903cfee54e7b81ab6e2fdfbf75b3de5bb9fb621e09ab81ad91"} Aug 04 19:16:52.865910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.865891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" event={"ID":"a5a17e53-02ff-463a-9919-160910ac2e10","Type":"ContainerStarted","Data":"a3726956b806b96c872032c484c456b9d8c6105300420beb8bccabb9f1712c83"} Aug 04 19:16:52.960420 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-out\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tmz\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-kube-api-access-w8tmz\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.960794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-web-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.961393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.960941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.961393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.961275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.961490 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:16:52.961398 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle podName:78f583ca-7bff-40a7-b036-eae5039b7e6b nodeName:}" failed. No retries permitted until 2026-08-04 19:16:53.461378584 +0000 UTC m=+74.431405093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "78f583ca-7bff-40a7-b036-eae5039b7e6b") : configmap references non-existent config key: ca-bundle.crt Aug 04 19:16:52.963720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.963664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-out\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.963826 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.963791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.964433 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.964412 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-web-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.964715 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.964689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-config-volume\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.964835 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.964818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.965044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.965029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.965167 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.965146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.965683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.965658 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/78f583ca-7bff-40a7-b036-eae5039b7e6b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.965784 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.965763 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:52.972131 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:52.972112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tmz\" (UniqueName: \"kubernetes.io/projected/78f583ca-7bff-40a7-b036-eae5039b7e6b-kube-api-access-w8tmz\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:53.469781 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:53.469743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:53.470591 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:53.470564 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78f583ca-7bff-40a7-b036-eae5039b7e6b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"78f583ca-7bff-40a7-b036-eae5039b7e6b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:53.695207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:53.695177 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 19:16:53.800435 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:53.800410 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ffd8f474-bcmbw" Aug 04 19:16:54.261546 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.261403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 19:16:54.267082 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:54.267035 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f583ca_7bff_40a7_b036_eae5039b7e6b.slice/crio-c1f4a4b4886093930ede4cd6ec564fcccd4a3c9f3c18a43a69554ff9340143f3 WatchSource:0}: Error finding container c1f4a4b4886093930ede4cd6ec564fcccd4a3c9f3c18a43a69554ff9340143f3: Status 404 returned error can't find the container with id c1f4a4b4886093930ede4cd6ec564fcccd4a3c9f3c18a43a69554ff9340143f3 Aug 04 19:16:54.874353 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.874321 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a85812-f966-4d91-bfe4-39a86252ea9a" containerID="b6f89552315570fc628f5da173afcf99b78e8c42eb4bc1b038539f69318db4fc" exitCode=0 Aug 04 19:16:54.874529 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.874407 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkppb" event={"ID":"08a85812-f966-4d91-bfe4-39a86252ea9a","Type":"ContainerDied","Data":"b6f89552315570fc628f5da173afcf99b78e8c42eb4bc1b038539f69318db4fc"} Aug 04 19:16:54.876424 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.876400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" event={"ID":"a5a17e53-02ff-463a-9919-160910ac2e10","Type":"ContainerStarted","Data":"cbf0dc861bc02b9946894b98e07a3f03be5cbfdf744d38ea724b1ff6a962bc30"} Aug 04 19:16:54.876509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.876454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" event={"ID":"a5a17e53-02ff-463a-9919-160910ac2e10","Type":"ContainerStarted","Data":"a829d1120d7befccda590c2bbc7d8e46db5b3598f0e644d36a04d865b7a1ea96"} Aug 04 19:16:54.876509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.876482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" event={"ID":"a5a17e53-02ff-463a-9919-160910ac2e10","Type":"ContainerStarted","Data":"dc68d90792d6affa09dc6a1c816d8d8a6ffaba2cd87c5dc12ce0ad1f315ad6db"} Aug 04 19:16:54.877750 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.877728 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"c1f4a4b4886093930ede4cd6ec564fcccd4a3c9f3c18a43a69554ff9340143f3"} Aug 04 19:16:54.911991 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:54.911941 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-q4gdn" podStartSLOduration=2.524684823 podStartE2EDuration="3.911921706s" totalImagesPullingTime="1.387236883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:54.910857574 +0000 UTC m=+75.880884094" watchObservedRunningTime="2026-08-04 19:16:54.911921706 +0000 UTC m=+75.881948227" Aug 04 19:16:55.882601 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:55.882569 2567 generic.go:356] "Generic (PLEG): container finished" podID="78f583ca-7bff-40a7-b036-eae5039b7e6b" containerID="3628beb5bb6b4119798a842b9a0a05a7284635db53e549e12256601867b0a06b" exitCode=0 Aug 04 19:16:55.882973 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:55.882650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerDied","Data":"3628beb5bb6b4119798a842b9a0a05a7284635db53e549e12256601867b0a06b"} Aug 04 19:16:55.884951 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:55.884924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkppb" event={"ID":"08a85812-f966-4d91-bfe4-39a86252ea9a","Type":"ContainerStarted","Data":"06dda1a904ed1ccecf5f953652d716ca723a11e125e8bcb06733717f9bef4cd9"} Aug 04 19:16:55.885077 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:55.884969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkppb" event={"ID":"08a85812-f966-4d91-bfe4-39a86252ea9a","Type":"ContainerStarted","Data":"5de50334c5accf583f8a51acf534ded210041d4f63a86a97f7812382f6459ab5"} Aug 04 19:16:55.932154 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:55.932109 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pkppb" podStartSLOduration=3.356916124 podStartE2EDuration="4.932096052s" totalImagesPullingTime="1.494573955s" totalInitContainerRuntime="80.605973ms" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:55.93095126 +0000 UTC m=+76.900977805" watchObservedRunningTime="2026-08-04 19:16:55.932096052 +0000 UTC m=+76.902122572" Aug 04 19:16:56.058378 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.058349 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5554d79c5f-k4j8w"] Aug 04 19:16:56.060333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.060313 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.062929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062908 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 19:16:56.063028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062955 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 19:16:56.063028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062914 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fk3bsr2tdbirm\"" Aug 04 19:16:56.063028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062936 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 19:16:56.063028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062940 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 19:16:56.063256 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.062942 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6ggn6\"" Aug 04 19:16:56.073626 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.073604 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5554d79c5f-k4j8w"] Aug 04 19:16:56.194677 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194625 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8z2n\" (UniqueName: \"kubernetes.io/projected/aa2e0347-ff51-4827-8107-d979e9b9c673-kube-api-access-c8z2n\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194677 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-tls\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194806 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa2e0347-ff51-4827-8107-d979e9b9c673-audit-log\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194806 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-client-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194806 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-client-certs\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194806 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-metrics-server-audit-profiles\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.194983 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.194822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.295953 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.295926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.295994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8z2n\" (UniqueName: \"kubernetes.io/projected/aa2e0347-ff51-4827-8107-d979e9b9c673-kube-api-access-c8z2n\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296025 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-tls\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296129 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa2e0347-ff51-4827-8107-d979e9b9c673-audit-log\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-client-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-client-certs\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296363 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-metrics-server-audit-profiles\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296562 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aa2e0347-ff51-4827-8107-d979e9b9c673-audit-log\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.296751 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.296719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.297140 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.297117 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aa2e0347-ff51-4827-8107-d979e9b9c673-metrics-server-audit-profiles\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.298699 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.298668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-client-ca-bundle\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.299170 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.299149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-client-certs\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.299308 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.299288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aa2e0347-ff51-4827-8107-d979e9b9c673-secret-metrics-server-tls\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.304479 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.304427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8z2n\" (UniqueName: \"kubernetes.io/projected/aa2e0347-ff51-4827-8107-d979e9b9c673-kube-api-access-c8z2n\") pod \"metrics-server-5554d79c5f-k4j8w\" (UID: \"aa2e0347-ff51-4827-8107-d979e9b9c673\") " pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.369146 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.369120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:16:56.439625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.439570 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:16:56.442714 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.442693 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.450497 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.450432 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 19:16:56.455183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.455029 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:16:56.516315 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.516291 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5554d79c5f-k4j8w"] Aug 04 19:16:56.519142 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:56.519109 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2e0347_ff51_4827_8107_d979e9b9c673.slice/crio-48d960825b1e4f497c9cdce3b28f653258032edd49dd2aa44b390dea19274cf0 WatchSource:0}: Error finding container 48d960825b1e4f497c9cdce3b28f653258032edd49dd2aa44b390dea19274cf0: Status 404 returned error can't find the container with id 48d960825b1e4f497c9cdce3b28f653258032edd49dd2aa44b390dea19274cf0 Aug 04 19:16:56.600282 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492pn\" (UniqueName: \"kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600373 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600373 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600330 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600373 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600548 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600548 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.600548 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.600543 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701729 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492pn\" (UniqueName: \"kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701729 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701733 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.701892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.701848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.702649 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.702627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.702737 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.702648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.702737 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.702688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.702808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.702767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.704227 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.704198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.704311 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.704287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.710504 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.710485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492pn\" (UniqueName: \"kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn\") pod \"console-649dcc6df4-pqlbf\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.756205 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.756170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:16:56.892529 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.892491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" event={"ID":"aa2e0347-ff51-4827-8107-d979e9b9c673","Type":"ContainerStarted","Data":"48d960825b1e4f497c9cdce3b28f653258032edd49dd2aa44b390dea19274cf0"} Aug 04 19:16:56.892949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:56.892908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:16:56.895381 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:56.895358 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe342a9b_3554_4fc1_9737_d43ea7af9401.slice/crio-a3385348dcdbf485cd0a66555e9cbe33414160529630d1646285f73f55b2d4c9 WatchSource:0}: Error finding container a3385348dcdbf485cd0a66555e9cbe33414160529630d1646285f73f55b2d4c9: Status 404 returned error can't find the container with id a3385348dcdbf485cd0a66555e9cbe33414160529630d1646285f73f55b2d4c9 Aug 04 19:16:57.105170 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.105136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:57.105307 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.105186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:57.105643 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.105610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc129f0a-75ee-4332-a393-60520f6c4ac7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:57.107354 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.107337 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc129f0a-75ee-4332-a393-60520f6c4ac7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pvgm\" (UID: \"bc129f0a-75ee-4332-a393-60520f6c4ac7\") " pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:57.214218 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.214192 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pvgm" Aug 04 19:16:57.379795 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.379727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pvgm"] Aug 04 19:16:57.668104 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:16:57.668007 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc129f0a_75ee_4332_a393_60520f6c4ac7.slice/crio-5329529f7d187f077c90529091a4c94b2a5e7e63f126390e23b6773030bc28d5 WatchSource:0}: Error finding container 5329529f7d187f077c90529091a4c94b2a5e7e63f126390e23b6773030bc28d5: Status 404 returned error can't find the container with id 5329529f7d187f077c90529091a4c94b2a5e7e63f126390e23b6773030bc28d5 Aug 04 19:16:57.898137 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.898100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pvgm" event={"ID":"bc129f0a-75ee-4332-a393-60520f6c4ac7","Type":"ContainerStarted","Data":"68ce6a96be79a9446f8e1fcff43af48046f3efeed464326b5b34b16a480c3737"} Aug 04 19:16:57.898452 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.898182 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pvgm" event={"ID":"bc129f0a-75ee-4332-a393-60520f6c4ac7","Type":"ContainerStarted","Data":"5329529f7d187f077c90529091a4c94b2a5e7e63f126390e23b6773030bc28d5"} Aug 04 19:16:57.900802 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.900779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"9f7d705f7359c94856daa2e0d249e1917daec08a5f4c56a60af15e889411a1af"} Aug 04 19:16:57.900911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.900849 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"715d6919d9ddd1641cd7a86284d453b8c4a449fa273d47802f917842ef13ed14"} Aug 04 19:16:57.900911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.900891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"51b6ed3800df0457fe0a7a72670d46976542631bfff0a2e93f4d3c6cd841f951"} Aug 04 19:16:57.902262 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.902241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649dcc6df4-pqlbf" event={"ID":"fe342a9b-3554-4fc1-9737-d43ea7af9401","Type":"ContainerStarted","Data":"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43"} Aug 04 19:16:57.902338 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.902277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649dcc6df4-pqlbf" event={"ID":"fe342a9b-3554-4fc1-9737-d43ea7af9401","Type":"ContainerStarted","Data":"a3385348dcdbf485cd0a66555e9cbe33414160529630d1646285f73f55b2d4c9"} Aug 04 19:16:57.931631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:57.931425 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-649dcc6df4-pqlbf" podStartSLOduration=1.9314053329999998 podStartE2EDuration="1.931405333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:57.930911601 +0000 UTC m=+78.900938123" watchObservedRunningTime="2026-08-04 19:16:57.931405333 +0000 UTC m=+78.901431856" Aug 04 19:16:58.911132 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:58.911084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"a4498e0b0df57313cc01203973b77a145ddc4271a32420e0a4ad18600dbf2386"} Aug 04 19:16:58.911403 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:58.911194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"ddb3f93384e7249a90f4960f9744d3bf004d1828fa4e0d3a7db315a5cb14579d"} Aug 04 19:16:59.916209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:59.916129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" event={"ID":"aa2e0347-ff51-4827-8107-d979e9b9c673","Type":"ContainerStarted","Data":"663ec0ae6b08ef43f8702466355504da1d8acfb7af48c31693287ac6a972cc34"} Aug 04 19:16:59.918014 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:59.917991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pvgm" event={"ID":"bc129f0a-75ee-4332-a393-60520f6c4ac7","Type":"ContainerStarted","Data":"79c0c20f922955c06f56c9952dca4ba6a0b95c2f83488a6f51bfe5ced4ec6523"} Aug 04 19:16:59.921148 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:59.921128 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"78f583ca-7bff-40a7-b036-eae5039b7e6b","Type":"ContainerStarted","Data":"95dfd2010e32ca41f119385d4abfe35a4721cc7ebb60acfb4297f0daf4862936"} Aug 04 19:16:59.932823 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:59.932784 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" podStartSLOduration=1.163147604 podStartE2EDuration="3.93276748s" totalImagesPullingTime="2.769619876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:59.932401253 +0000 UTC m=+80.902427790" watchObservedRunningTime="2026-08-04 19:16:59.93276748 +0000 UTC m=+80.902794000" Aug 04 19:16:59.959770 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:16:59.959728 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.101335161 podStartE2EDuration="7.95971412s" totalImagesPullingTime="3.839668227s" totalInitContainerRuntime="18.710732ms" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:59.957304175 +0000 UTC m=+80.927330725" watchObservedRunningTime="2026-08-04 19:16:59.95971412 +0000 UTC m=+80.929740639" Aug 04 19:17:01.930469 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:01.930433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pvgm" event={"ID":"bc129f0a-75ee-4332-a393-60520f6c4ac7","Type":"ContainerStarted","Data":"a44a7fc7b7dfd332f372451274cd844573e46b6708d5b45600834f8067dd049d"} Aug 04 19:17:01.949805 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:01.949753 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6pvgm" podStartSLOduration=17.734365508 podStartE2EDuration="20.949734855s" totalImagesPullingTime="3.215369347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:01.949139183 +0000 UTC m=+82.919165704" watchObservedRunningTime="2026-08-04 19:17:01.949734855 +0000 UTC m=+82.919761375" Aug 04 19:17:06.757322 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:06.757291 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:17:06.757758 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:06.757352 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:17:06.761949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:06.761927 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:17:06.955170 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:06.955140 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:17:14.006436 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.006394 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-585fc4cc99-4s4xr" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerName="console" containerID="cri-o://16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21" gracePeriod=15 Aug 04 19:17:14.234754 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.234732 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:17:14.334355 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334334 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334464 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w88p9\" (UniqueName: \"kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334464 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334407 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334547 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334460 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334547 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334490 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334702 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334679 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config\") pod \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\" (UID: \"9b0a97f2-586c-4b34-8e1a-975cf45fe36b\") " Aug 04 19:17:14.334777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334762 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config" (OuterVolumeSpecName: "console-config") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:14.334864 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334839 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:14.334919 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334844 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:14.334960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334934 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-oauth-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.334960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334949 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.334960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.334958 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.336777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.336752 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:14.336870 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.336795 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9" (OuterVolumeSpecName: "kube-api-access-w88p9") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "kube-api-access-w88p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:17:14.336870 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.336819 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b0a97f2-586c-4b34-8e1a-975cf45fe36b" (UID: "9b0a97f2-586c-4b34-8e1a-975cf45fe36b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:14.435780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.435738 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w88p9\" (UniqueName: \"kubernetes.io/projected/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-kube-api-access-w88p9\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.435780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.435776 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.435900 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.435786 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b0a97f2-586c-4b34-8e1a-975cf45fe36b-console-oauth-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:17:14.980906 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.980875 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerID="16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21" exitCode=0 Aug 04 19:17:14.981084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.980916 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fc4cc99-4s4xr" Aug 04 19:17:14.981084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.980949 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fc4cc99-4s4xr" event={"ID":"9b0a97f2-586c-4b34-8e1a-975cf45fe36b","Type":"ContainerDied","Data":"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21"} Aug 04 19:17:14.981084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.980986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fc4cc99-4s4xr" event={"ID":"9b0a97f2-586c-4b34-8e1a-975cf45fe36b","Type":"ContainerDied","Data":"6d599a825641aa65bf3c4263dde82c488d99d9252ea73e912568715270831acf"} Aug 04 19:17:14.981084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.981018 2567 scope.go:117] "RemoveContainer" containerID="16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21" Aug 04 19:17:14.989295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.989279 2567 scope.go:117] "RemoveContainer" containerID="16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21" Aug 04 19:17:14.989557 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:17:14.989535 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21\": container with ID starting with 16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21 not found: ID does not exist" containerID="16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21" Aug 04 19:17:14.989604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:14.989565 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21"} err="failed to get container status \"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21\": rpc error: code = NotFound desc = could not find container \"16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21\": container with ID starting with 16029312d18ea3c6cee534601e1c574f97dd9729682ffa9c891edd8302a0ad21 not found: ID does not exist" Aug 04 19:17:15.001208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:15.001189 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:17:15.008548 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:15.008529 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-585fc4cc99-4s4xr"] Aug 04 19:17:15.503204 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:15.503175 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" path="/var/lib/kubelet/pods/9b0a97f2-586c-4b34-8e1a-975cf45fe36b/volumes" Aug 04 19:17:16.369437 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:16.369410 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:17:16.369775 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:16.369462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:17:20.860777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:20.860746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-62hbz" Aug 04 19:17:36.375220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:36.375186 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:17:36.378999 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:17:36.378973 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5554d79c5f-k4j8w" Aug 04 19:18:16.065607 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.065567 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f9f54f686-rp52c"] Aug 04 19:18:16.066220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.065938 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerName="console" Aug 04 19:18:16.066220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.065956 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerName="console" Aug 04 19:18:16.066220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.066028 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0a97f2-586c-4b34-8e1a-975cf45fe36b" containerName="console" Aug 04 19:18:16.069393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.069372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.072122 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072098 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 19:18:16.072226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072160 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 19:18:16.072226 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072161 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 19:18:16.072357 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072338 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 19:18:16.072453 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072436 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-92wzh\"" Aug 04 19:18:16.072530 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.072515 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 19:18:16.077165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.077145 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 19:18:16.097380 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.097359 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9f54f686-rp52c"] Aug 04 19:18:16.178195 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178313 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-metrics-client-ca\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178313 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-serving-certs-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178313 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178467 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178467 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7mz\" (UniqueName: \"kubernetes.io/projected/f384657b-1743-4c2e-b3fc-635dad005b56-kube-api-access-vh7mz\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178467 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-federate-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.178587 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.178477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279494 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-federate-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279588 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279588 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279530 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279588 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-metrics-client-ca\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279722 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279587 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-serving-certs-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279722 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279722 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.279928 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.279777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7mz\" (UniqueName: \"kubernetes.io/projected/f384657b-1743-4c2e-b3fc-635dad005b56-kube-api-access-vh7mz\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.280463 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.280442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-serving-certs-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.280604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.280490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.280743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.280717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f384657b-1743-4c2e-b3fc-635dad005b56-metrics-client-ca\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.282173 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.282151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.282252 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.282176 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-telemeter-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.282299 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.282286 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-federate-client-tls\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.282400 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.282377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f384657b-1743-4c2e-b3fc-635dad005b56-secret-telemeter-client\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.287707 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.287686 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7mz\" (UniqueName: \"kubernetes.io/projected/f384657b-1743-4c2e-b3fc-635dad005b56-kube-api-access-vh7mz\") pod \"telemeter-client-7f9f54f686-rp52c\" (UID: \"f384657b-1743-4c2e-b3fc-635dad005b56\") " pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.378629 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.378580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" Aug 04 19:18:16.506550 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:16.506511 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9f54f686-rp52c"] Aug 04 19:18:16.508583 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:18:16.508558 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf384657b_1743_4c2e_b3fc_635dad005b56.slice/crio-3606468e837cfcb0014c09788a676c2b2ff402ac30da956d13b29f565b627a36 WatchSource:0}: Error finding container 3606468e837cfcb0014c09788a676c2b2ff402ac30da956d13b29f565b627a36: Status 404 returned error can't find the container with id 3606468e837cfcb0014c09788a676c2b2ff402ac30da956d13b29f565b627a36 Aug 04 19:18:17.194727 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:17.194691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" event={"ID":"f384657b-1743-4c2e-b3fc-635dad005b56","Type":"ContainerStarted","Data":"3606468e837cfcb0014c09788a676c2b2ff402ac30da956d13b29f565b627a36"} Aug 04 19:18:18.200561 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.200523 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" event={"ID":"f384657b-1743-4c2e-b3fc-635dad005b56","Type":"ContainerStarted","Data":"005a1879ac7a039f6ba435a45e7101bab31b6c1f723883b228dd6e85f167c8f3"} Aug 04 19:18:18.200889 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.200602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" event={"ID":"f384657b-1743-4c2e-b3fc-635dad005b56","Type":"ContainerStarted","Data":"e63f278577c994d55c744d8041b5cea31830bc116ec0862c7bab5f7f9fa70278"} Aug 04 19:18:18.200889 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.200634 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" event={"ID":"f384657b-1743-4c2e-b3fc-635dad005b56","Type":"ContainerStarted","Data":"bf6dee7987539f59782fdce34a748ea41f65e0f4b7b59b6774a3256019335823"} Aug 04 19:18:18.227982 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.227933 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f9f54f686-rp52c" podStartSLOduration=0.716492793 podStartE2EDuration="2.227918414s" totalImagesPullingTime="1.511425621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:18:18.222950382 +0000 UTC m=+159.192976902" watchObservedRunningTime="2026-08-04 19:18:18.227918414 +0000 UTC m=+159.197944933" Aug 04 19:18:18.299213 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.299186 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:18:18.302272 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.302252 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.319576 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.319557 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:18:18.396583 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396583 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396583 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396572 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcl7\" (UniqueName: \"kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.396968 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.396915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497214 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497334 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497224 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497500 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497500 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcl7\" (UniqueName: \"kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497500 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.497914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.498022 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.497999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.498120 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.498099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.498225 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.498207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.499669 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.499650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.499816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.499799 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.505194 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.505176 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcl7\" (UniqueName: \"kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7\") pod \"console-869759c8cd-8xpbp\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.611258 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.611235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:18.730408 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:18.730386 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:18:18.732735 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:18:18.732704 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b302b2_3395_49d3_8b85_472a93512ecd.slice/crio-9f52363e34c3464b89d0fa255d7cc3c55bae23cebf77c4a06a06c8b4107d461d WatchSource:0}: Error finding container 9f52363e34c3464b89d0fa255d7cc3c55bae23cebf77c4a06a06c8b4107d461d: Status 404 returned error can't find the container with id 9f52363e34c3464b89d0fa255d7cc3c55bae23cebf77c4a06a06c8b4107d461d Aug 04 19:18:19.205469 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:19.205412 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869759c8cd-8xpbp" event={"ID":"46b302b2-3395-49d3-8b85-472a93512ecd","Type":"ContainerStarted","Data":"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208"} Aug 04 19:18:19.205847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:19.205481 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869759c8cd-8xpbp" event={"ID":"46b302b2-3395-49d3-8b85-472a93512ecd","Type":"ContainerStarted","Data":"9f52363e34c3464b89d0fa255d7cc3c55bae23cebf77c4a06a06c8b4107d461d"} Aug 04 19:18:19.222390 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:19.222343 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-869759c8cd-8xpbp" podStartSLOduration=1.222328572 podStartE2EDuration="1.222328572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:18:19.221341761 +0000 UTC m=+160.191368280" watchObservedRunningTime="2026-08-04 19:18:19.222328572 +0000 UTC m=+160.192355092" Aug 04 19:18:28.612290 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:28.612247 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:28.612727 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:28.612312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:28.616916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:28.616895 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:29.244162 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.244134 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:18:29.292895 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.292859 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:18:29.373675 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.373648 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:18:29.378780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.378757 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.387683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.387663 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:18:29.473905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.473875 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.473905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.473907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.474193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.473988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.474193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.474018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.474193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.474070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.474193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.474117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.474193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.474132 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsgt\" (UniqueName: \"kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574569 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsgt\" (UniqueName: \"kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574786 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574819 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574856 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.574901 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.574866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.575355 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.575331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.575591 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.575569 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.575660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.575610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.575660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.575638 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.576984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.576965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.577156 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.577140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.582694 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.582670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsgt\" (UniqueName: \"kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt\") pod \"console-59bcc48bdd-5xwrd\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.688832 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.688795 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:29.813935 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:29.813915 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:18:29.815930 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:18:29.815906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0858f19c_9f97_4d00_ba11_ded087643873.slice/crio-afedbdac8651a5405244bc2dcf830dfdbc33b527132005f249a5296c3e86fa4c WatchSource:0}: Error finding container afedbdac8651a5405244bc2dcf830dfdbc33b527132005f249a5296c3e86fa4c: Status 404 returned error can't find the container with id afedbdac8651a5405244bc2dcf830dfdbc33b527132005f249a5296c3e86fa4c Aug 04 19:18:30.244813 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:30.244719 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bcc48bdd-5xwrd" event={"ID":"0858f19c-9f97-4d00-ba11-ded087643873","Type":"ContainerStarted","Data":"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca"} Aug 04 19:18:30.244948 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:30.244814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bcc48bdd-5xwrd" event={"ID":"0858f19c-9f97-4d00-ba11-ded087643873","Type":"ContainerStarted","Data":"afedbdac8651a5405244bc2dcf830dfdbc33b527132005f249a5296c3e86fa4c"} Aug 04 19:18:30.265290 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:30.265246 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59bcc48bdd-5xwrd" podStartSLOduration=1.265232422 podStartE2EDuration="1.265232422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:18:30.263204818 +0000 UTC m=+171.233231337" watchObservedRunningTime="2026-08-04 19:18:30.265232422 +0000 UTC m=+171.235258940" Aug 04 19:18:39.689460 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:39.689376 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:39.689460 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:39.689438 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:39.694209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:39.694188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:40.286367 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:40.286339 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:18:40.333451 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:40.333423 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:18:54.317254 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.317203 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-649dcc6df4-pqlbf" podUID="fe342a9b-3554-4fc1-9737-d43ea7af9401" containerName="console" containerID="cri-o://cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43" gracePeriod=15 Aug 04 19:18:54.548635 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.548606 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:18:54.677693 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677601 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677693 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677648 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677703 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677732 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-492pn\" (UniqueName: \"kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677796 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.677908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.677833 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert\") pod \"fe342a9b-3554-4fc1-9737-d43ea7af9401\" (UID: \"fe342a9b-3554-4fc1-9737-d43ea7af9401\") " Aug 04 19:18:54.678197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.678100 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:18:54.678197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.678174 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config" (OuterVolumeSpecName: "console-config") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:18:54.678197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.678189 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:18:54.678405 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.678377 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:18:54.679939 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.679915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:18:54.680018 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.679937 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn" (OuterVolumeSpecName: "kube-api-access-492pn") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "kube-api-access-492pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:18:54.680018 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.679967 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe342a9b-3554-4fc1-9737-d43ea7af9401" (UID: "fe342a9b-3554-4fc1-9737-d43ea7af9401"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:18:54.778752 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778717 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.778752 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778746 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-oauth-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.778752 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778756 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.778752 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778764 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-trusted-ca-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.779012 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778773 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492pn\" (UniqueName: \"kubernetes.io/projected/fe342a9b-3554-4fc1-9737-d43ea7af9401-kube-api-access-492pn\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.779012 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778781 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe342a9b-3554-4fc1-9737-d43ea7af9401-console-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:54.779012 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:54.778791 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe342a9b-3554-4fc1-9737-d43ea7af9401-oauth-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:18:55.336707 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.336669 2567 generic.go:356] "Generic (PLEG): container finished" podID="fe342a9b-3554-4fc1-9737-d43ea7af9401" containerID="cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43" exitCode=0 Aug 04 19:18:55.337190 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.336729 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649dcc6df4-pqlbf" Aug 04 19:18:55.337190 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.336734 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649dcc6df4-pqlbf" event={"ID":"fe342a9b-3554-4fc1-9737-d43ea7af9401","Type":"ContainerDied","Data":"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43"} Aug 04 19:18:55.337190 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.336816 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649dcc6df4-pqlbf" event={"ID":"fe342a9b-3554-4fc1-9737-d43ea7af9401","Type":"ContainerDied","Data":"a3385348dcdbf485cd0a66555e9cbe33414160529630d1646285f73f55b2d4c9"} Aug 04 19:18:55.337190 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.336849 2567 scope.go:117] "RemoveContainer" containerID="cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43" Aug 04 19:18:55.344916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.344898 2567 scope.go:117] "RemoveContainer" containerID="cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43" Aug 04 19:18:55.345192 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:18:55.345168 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43\": container with ID starting with cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43 not found: ID does not exist" containerID="cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43" Aug 04 19:18:55.345274 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.345197 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43"} err="failed to get container status \"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43\": rpc error: code = NotFound desc = could not find container \"cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43\": container with ID starting with cf2e1250a9d0c362e355128809163a23f9c0970b0277fef80091f7963c36da43 not found: ID does not exist" Aug 04 19:18:55.356721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.356698 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:18:55.358533 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.358513 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-649dcc6df4-pqlbf"] Aug 04 19:18:55.503364 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:18:55.503329 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe342a9b-3554-4fc1-9737-d43ea7af9401" path="/var/lib/kubelet/pods/fe342a9b-3554-4fc1-9737-d43ea7af9401/volumes" Aug 04 19:19:05.356129 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.356069 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-869759c8cd-8xpbp" podUID="46b302b2-3395-49d3-8b85-472a93512ecd" containerName="console" containerID="cri-o://4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208" gracePeriod=15 Aug 04 19:19:05.593311 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.593290 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:19:05.764916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.764839 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.764916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.764877 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.764916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.764894 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.765183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.764939 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.765183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.764970 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.765183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whcl7\" (UniqueName: \"kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.765183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765087 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca\") pod \"46b302b2-3395-49d3-8b85-472a93512ecd\" (UID: \"46b302b2-3395-49d3-8b85-472a93512ecd\") " Aug 04 19:19:05.765406 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765380 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:19:05.765474 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765380 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:19:05.765527 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765500 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config" (OuterVolumeSpecName: "console-config") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:19:05.765574 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.765540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca" (OuterVolumeSpecName: "service-ca") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:19:05.767181 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.767150 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:19:05.767276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.767183 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:19:05.767385 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.767371 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7" (OuterVolumeSpecName: "kube-api-access-whcl7") pod "46b302b2-3395-49d3-8b85-472a93512ecd" (UID: "46b302b2-3395-49d3-8b85-472a93512ecd"). InnerVolumeSpecName "kube-api-access-whcl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:19:05.865891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865845 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.865891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865887 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-trusted-ca-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.865891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865897 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-oauth-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.865891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865906 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-oauth-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.866174 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865914 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46b302b2-3395-49d3-8b85-472a93512ecd-console-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.866174 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865922 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46b302b2-3395-49d3-8b85-472a93512ecd-console-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:05.866174 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:05.865931 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcl7\" (UniqueName: \"kubernetes.io/projected/46b302b2-3395-49d3-8b85-472a93512ecd-kube-api-access-whcl7\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:06.377361 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.377322 2567 generic.go:356] "Generic (PLEG): container finished" podID="46b302b2-3395-49d3-8b85-472a93512ecd" containerID="4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208" exitCode=0 Aug 04 19:19:06.377743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.377380 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869759c8cd-8xpbp" Aug 04 19:19:06.377743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.377389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869759c8cd-8xpbp" event={"ID":"46b302b2-3395-49d3-8b85-472a93512ecd","Type":"ContainerDied","Data":"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208"} Aug 04 19:19:06.377743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.377466 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869759c8cd-8xpbp" event={"ID":"46b302b2-3395-49d3-8b85-472a93512ecd","Type":"ContainerDied","Data":"9f52363e34c3464b89d0fa255d7cc3c55bae23cebf77c4a06a06c8b4107d461d"} Aug 04 19:19:06.377743 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.377492 2567 scope.go:117] "RemoveContainer" containerID="4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208" Aug 04 19:19:06.385858 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.385843 2567 scope.go:117] "RemoveContainer" containerID="4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208" Aug 04 19:19:06.386116 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:19:06.386094 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208\": container with ID starting with 4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208 not found: ID does not exist" containerID="4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208" Aug 04 19:19:06.386208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.386123 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208"} err="failed to get container status \"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208\": rpc error: code = NotFound desc = could not find container \"4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208\": container with ID starting with 4717b08efd878149b19e6af9f68b95009a88c1a9488e6b297b3d6a76a06f9208 not found: ID does not exist" Aug 04 19:19:06.398379 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.398356 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:19:06.401891 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:06.401871 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-869759c8cd-8xpbp"] Aug 04 19:19:07.502905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:07.502872 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b302b2-3395-49d3-8b85-472a93512ecd" path="/var/lib/kubelet/pods/46b302b2-3395-49d3-8b85-472a93512ecd/volumes" Aug 04 19:19:11.071340 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071299 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt"] Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071621 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b302b2-3395-49d3-8b85-472a93512ecd" containerName="console" Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071634 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b302b2-3395-49d3-8b85-472a93512ecd" containerName="console" Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071643 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe342a9b-3554-4fc1-9737-d43ea7af9401" containerName="console" Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071649 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe342a9b-3554-4fc1-9737-d43ea7af9401" containerName="console" Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071695 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b302b2-3395-49d3-8b85-472a93512ecd" containerName="console" Aug 04 19:19:11.071710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.071705 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe342a9b-3554-4fc1-9737-d43ea7af9401" containerName="console" Aug 04 19:19:11.076122 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.076102 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.078883 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.078859 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f9hlb\"" Aug 04 19:19:11.078973 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.078934 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 19:19:11.078973 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.078943 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 19:19:11.087134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.087115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt"] Aug 04 19:19:11.204581 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.204554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xf6\" (UniqueName: \"kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.204699 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.204600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.204699 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.204675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.305287 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.305255 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.305411 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.305314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xf6\" (UniqueName: \"kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.305411 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.305366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.305657 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.305596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.305738 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.305660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.314803 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.314776 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xf6\" (UniqueName: \"kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.384905 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.384850 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:11.507845 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:11.507826 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt"] Aug 04 19:19:11.510091 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:19:11.510044 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc238cea7_6527_4a9c_a076_af586918a63c.slice/crio-06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921 WatchSource:0}: Error finding container 06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921: Status 404 returned error can't find the container with id 06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921 Aug 04 19:19:12.401348 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:12.401310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" event={"ID":"c238cea7-6527-4a9c-a076-af586918a63c","Type":"ContainerStarted","Data":"06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921"} Aug 04 19:19:17.421847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:17.421817 2567 generic.go:356] "Generic (PLEG): container finished" podID="c238cea7-6527-4a9c-a076-af586918a63c" containerID="f5852a7b94b49596e9a816cce046a1624772295ccdcde8807a3a027ecd6a2e6b" exitCode=0 Aug 04 19:19:17.422230 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:17.421895 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" event={"ID":"c238cea7-6527-4a9c-a076-af586918a63c","Type":"ContainerDied","Data":"f5852a7b94b49596e9a816cce046a1624772295ccdcde8807a3a027ecd6a2e6b"} Aug 04 19:19:31.476147 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:31.476034 2567 generic.go:356] "Generic (PLEG): container finished" podID="c238cea7-6527-4a9c-a076-af586918a63c" containerID="61cb3a6f95f39069a4ee340dd4ce55d1114fb802047b32f4c0334ab9fd8aab7b" exitCode=0 Aug 04 19:19:31.476147 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:31.476116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" event={"ID":"c238cea7-6527-4a9c-a076-af586918a63c","Type":"ContainerDied","Data":"61cb3a6f95f39069a4ee340dd4ce55d1114fb802047b32f4c0334ab9fd8aab7b"} Aug 04 19:19:38.512783 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:38.512712 2567 generic.go:356] "Generic (PLEG): container finished" podID="c238cea7-6527-4a9c-a076-af586918a63c" containerID="c629554b64f87fe2e4143d2ce9dd448da3fe460617c6cab7a65febb6421da50a" exitCode=0 Aug 04 19:19:38.513159 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:38.512795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" event={"ID":"c238cea7-6527-4a9c-a076-af586918a63c","Type":"ContainerDied","Data":"c629554b64f87fe2e4143d2ce9dd448da3fe460617c6cab7a65febb6421da50a"} Aug 04 19:19:39.645335 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.645280 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:39.726862 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.726832 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle\") pod \"c238cea7-6527-4a9c-a076-af586918a63c\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " Aug 04 19:19:39.726862 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.726866 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util\") pod \"c238cea7-6527-4a9c-a076-af586918a63c\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " Aug 04 19:19:39.727070 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.726889 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8xf6\" (UniqueName: \"kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6\") pod \"c238cea7-6527-4a9c-a076-af586918a63c\" (UID: \"c238cea7-6527-4a9c-a076-af586918a63c\") " Aug 04 19:19:39.727556 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.727528 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle" (OuterVolumeSpecName: "bundle") pod "c238cea7-6527-4a9c-a076-af586918a63c" (UID: "c238cea7-6527-4a9c-a076-af586918a63c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:19:39.729010 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.728985 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6" (OuterVolumeSpecName: "kube-api-access-v8xf6") pod "c238cea7-6527-4a9c-a076-af586918a63c" (UID: "c238cea7-6527-4a9c-a076-af586918a63c"). InnerVolumeSpecName "kube-api-access-v8xf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:19:39.731092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.731067 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util" (OuterVolumeSpecName: "util") pod "c238cea7-6527-4a9c-a076-af586918a63c" (UID: "c238cea7-6527-4a9c-a076-af586918a63c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:19:39.827561 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.827526 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:39.827561 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.827559 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c238cea7-6527-4a9c-a076-af586918a63c-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:39.827561 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:39.827568 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xf6\" (UniqueName: \"kubernetes.io/projected/c238cea7-6527-4a9c-a076-af586918a63c-kube-api-access-v8xf6\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:40.521298 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:40.521259 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" event={"ID":"c238cea7-6527-4a9c-a076-af586918a63c","Type":"ContainerDied","Data":"06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921"} Aug 04 19:19:40.521298 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:40.521304 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d1275e16fc45d3a2f72ad8e2d8bfed01e9c29c7d97945d7122e5a5ab892921" Aug 04 19:19:40.521494 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:40.521270 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbjjt" Aug 04 19:19:44.207718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.207687 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c"] Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208073 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="extract" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208090 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="extract" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208116 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="util" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208124 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="util" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208137 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="pull" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208146 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="pull" Aug 04 19:19:44.208431 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.208222 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c238cea7-6527-4a9c-a076-af586918a63c" containerName="extract" Aug 04 19:19:44.210366 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.210341 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.213064 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.213032 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wlzpv\"" Aug 04 19:19:44.213185 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.213063 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:19:44.213185 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.213097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 19:19:44.228239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.228219 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c"] Aug 04 19:19:44.260917 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.260893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fsw\" (UniqueName: \"kubernetes.io/projected/164dd777-f042-47d6-96b9-447cdb0ee1fa-kube-api-access-g2fsw\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.261093 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.260991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164dd777-f042-47d6-96b9-447cdb0ee1fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.361555 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.361526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164dd777-f042-47d6-96b9-447cdb0ee1fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.361672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.361561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fsw\" (UniqueName: \"kubernetes.io/projected/164dd777-f042-47d6-96b9-447cdb0ee1fa-kube-api-access-g2fsw\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.361894 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.361877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164dd777-f042-47d6-96b9-447cdb0ee1fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.373498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.373462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fsw\" (UniqueName: \"kubernetes.io/projected/164dd777-f042-47d6-96b9-447cdb0ee1fa-kube-api-access-g2fsw\") pod \"cert-manager-operator-controller-manager-55c444bf86-blt9c\" (UID: \"164dd777-f042-47d6-96b9-447cdb0ee1fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.519383 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.519320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" Aug 04 19:19:44.645657 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:44.645623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c"] Aug 04 19:19:44.648711 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:19:44.648684 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164dd777_f042_47d6_96b9_447cdb0ee1fa.slice/crio-49323fdf2d7dbeef3201a083d9d0be667b31a91bef9608892e2ea882dc4472f8 WatchSource:0}: Error finding container 49323fdf2d7dbeef3201a083d9d0be667b31a91bef9608892e2ea882dc4472f8: Status 404 returned error can't find the container with id 49323fdf2d7dbeef3201a083d9d0be667b31a91bef9608892e2ea882dc4472f8 Aug 04 19:19:45.545064 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:45.545012 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" event={"ID":"164dd777-f042-47d6-96b9-447cdb0ee1fa","Type":"ContainerStarted","Data":"49323fdf2d7dbeef3201a083d9d0be667b31a91bef9608892e2ea882dc4472f8"} Aug 04 19:19:46.551189 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:46.551150 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" event={"ID":"164dd777-f042-47d6-96b9-447cdb0ee1fa","Type":"ContainerStarted","Data":"3bc11f3a8b169313d11d85b7edc8e644e12dc76e3127382e09b0910996fa2420"} Aug 04 19:19:46.574335 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:46.574296 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-blt9c" podStartSLOduration=1.076250545 podStartE2EDuration="2.574282152s" totalImagesPullingTime="1.498031607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:19:46.573941775 +0000 UTC m=+247.543968285" watchObservedRunningTime="2026-08-04 19:19:46.574282152 +0000 UTC m=+247.544308671" Aug 04 19:19:47.844414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.844387 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj"] Aug 04 19:19:47.846823 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.846802 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:47.849311 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.849285 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f9hlb\"" Aug 04 19:19:47.849672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.849657 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 19:19:47.850339 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.850323 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 19:19:47.862693 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.862670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj"] Aug 04 19:19:47.996067 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.996026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:47.996189 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.996103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm8r\" (UniqueName: \"kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:47.996189 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:47.996131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.067374 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.067344 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vp2zc"] Aug 04 19:19:48.070213 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.070194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.078807 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.078788 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 19:19:48.078923 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.078874 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 19:19:48.078923 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.078908 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xprvl\"" Aug 04 19:19:48.090111 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.090094 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vp2zc"] Aug 04 19:19:48.096927 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.096874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.096927 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.096914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm8r\" (UniqueName: \"kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.097104 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.096941 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.097309 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.097288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.097376 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.097317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.115286 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.115259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm8r\" (UniqueName: \"kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.156245 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.156206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:48.197336 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.197310 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.197452 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.197390 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5xv\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-kube-api-access-hk5xv\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.292930 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.292901 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj"] Aug 04 19:19:48.295992 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:19:48.295968 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb689ac7_550e_4e9e_8ba5_022d04ba0aaf.slice/crio-b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b WatchSource:0}: Error finding container b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b: Status 404 returned error can't find the container with id b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b Aug 04 19:19:48.297768 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.297746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5xv\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-kube-api-access-hk5xv\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.297871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.297824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.308277 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.308256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.311706 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.311687 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5xv\" (UniqueName: \"kubernetes.io/projected/bf17bc3e-25b1-4ff0-b09d-00759e9e42dc-kube-api-access-hk5xv\") pod \"cert-manager-webhook-6486ffd997-vp2zc\" (UID: \"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.394513 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.394493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:48.513797 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.513772 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vp2zc"] Aug 04 19:19:48.515553 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:19:48.515526 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf17bc3e_25b1_4ff0_b09d_00759e9e42dc.slice/crio-b3525f990f0d1ac50b18ac1664aee16ca956bd391c78615a82f271688dbd13c4 WatchSource:0}: Error finding container b3525f990f0d1ac50b18ac1664aee16ca956bd391c78615a82f271688dbd13c4: Status 404 returned error can't find the container with id b3525f990f0d1ac50b18ac1664aee16ca956bd391c78615a82f271688dbd13c4 Aug 04 19:19:48.559254 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.559223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" event={"ID":"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc","Type":"ContainerStarted","Data":"b3525f990f0d1ac50b18ac1664aee16ca956bd391c78615a82f271688dbd13c4"} Aug 04 19:19:48.560603 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.560572 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerID="a3c04b32dcb234d30b8cb53001c91f4f4720399b3b01bf2cfac169ad370d2574" exitCode=0 Aug 04 19:19:48.560688 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.560629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" event={"ID":"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf","Type":"ContainerDied","Data":"a3c04b32dcb234d30b8cb53001c91f4f4720399b3b01bf2cfac169ad370d2574"} Aug 04 19:19:48.560688 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:48.560679 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" event={"ID":"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf","Type":"ContainerStarted","Data":"b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b"} Aug 04 19:19:50.324972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.324945 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm"] Aug 04 19:19:50.330374 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.330322 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.332817 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.332795 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6fprv\"" Aug 04 19:19:50.338663 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.338638 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm"] Aug 04 19:19:50.413738 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.413712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.413850 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.413760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4892m\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-kube-api-access-4892m\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.514715 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.514678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.514881 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.514743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4892m\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-kube-api-access-4892m\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.525565 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.525537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.525696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.525680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4892m\" (UniqueName: \"kubernetes.io/projected/6584bda1-3ee9-4d3e-a042-0b1f1537a0b6-kube-api-access-4892m\") pod \"cert-manager-cainjector-559cdd8c87-jlbqm\" (UID: \"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.641863 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.641784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" Aug 04 19:19:50.925325 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:50.925298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm"] Aug 04 19:19:50.926770 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:19:50.926746 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6584bda1_3ee9_4d3e_a042_0b1f1537a0b6.slice/crio-e0782a1e0c912265553b9e5d55f86b8296c0b5bf15782a5ba94ef00abf2a218c WatchSource:0}: Error finding container e0782a1e0c912265553b9e5d55f86b8296c0b5bf15782a5ba94ef00abf2a218c: Status 404 returned error can't find the container with id e0782a1e0c912265553b9e5d55f86b8296c0b5bf15782a5ba94ef00abf2a218c Aug 04 19:19:51.575450 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:51.575418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" event={"ID":"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6","Type":"ContainerStarted","Data":"e0782a1e0c912265553b9e5d55f86b8296c0b5bf15782a5ba94ef00abf2a218c"} Aug 04 19:19:51.577086 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:51.577062 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerID="a4f6b8484e3004b3db29dca2aeeccb31f649eb5bdd5e26b118b63fea5c1e5b89" exitCode=0 Aug 04 19:19:51.577193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:51.577129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" event={"ID":"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf","Type":"ContainerDied","Data":"a4f6b8484e3004b3db29dca2aeeccb31f649eb5bdd5e26b118b63fea5c1e5b89"} Aug 04 19:19:52.583453 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:52.583415 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerID="47cfbd95a8539b0c10b2aadbc3551bb4f711a422c303b334e45ef06abc72c9b0" exitCode=0 Aug 04 19:19:52.583882 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:52.583492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" event={"ID":"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf","Type":"ContainerDied","Data":"47cfbd95a8539b0c10b2aadbc3551bb4f711a422c303b334e45ef06abc72c9b0"} Aug 04 19:19:54.453700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.453675 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:54.548531 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.548509 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util\") pod \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " Aug 04 19:19:54.548627 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.548546 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle\") pod \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " Aug 04 19:19:54.548627 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.548568 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqm8r\" (UniqueName: \"kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r\") pod \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\" (UID: \"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf\") " Aug 04 19:19:54.548940 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.548909 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle" (OuterVolumeSpecName: "bundle") pod "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" (UID: "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:19:54.550473 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.550448 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r" (OuterVolumeSpecName: "kube-api-access-bqm8r") pod "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" (UID: "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf"). InnerVolumeSpecName "kube-api-access-bqm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:19:54.552861 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.552835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util" (OuterVolumeSpecName: "util") pod "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" (UID: "fb689ac7-550e-4e9e-8ba5-022d04ba0aaf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:19:54.592972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.592933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" event={"ID":"6584bda1-3ee9-4d3e-a042-0b1f1537a0b6","Type":"ContainerStarted","Data":"ebdf765dcf4989ac96d6d9bd599e458b6e5d018409b04af0488a09f3864e8d5f"} Aug 04 19:19:54.595125 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.595102 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" event={"ID":"fb689ac7-550e-4e9e-8ba5-022d04ba0aaf","Type":"ContainerDied","Data":"b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b"} Aug 04 19:19:54.595212 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.595124 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgrrvj" Aug 04 19:19:54.595212 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.595147 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b798b856ee36e628ea95e927b54fbb9718819f5309b8e9c1f2d69fc0a2b96f1b" Aug 04 19:19:54.608546 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.608502 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jlbqm" podStartSLOduration=1.061218679 podStartE2EDuration="4.608488978s" totalImagesPullingTime="3.547270299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:19:54.607889123 +0000 UTC m=+255.577915642" watchObservedRunningTime="2026-08-04 19:19:54.608488978 +0000 UTC m=+255.578515496" Aug 04 19:19:54.649620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.649539 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:54.649620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.649573 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqm8r\" (UniqueName: \"kubernetes.io/projected/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-kube-api-access-bqm8r\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:54.649620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:54.649588 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb689ac7-550e-4e9e-8ba5-022d04ba0aaf-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:19:58.611786 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:58.611747 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" event={"ID":"bf17bc3e-25b1-4ff0-b09d-00759e9e42dc","Type":"ContainerStarted","Data":"cdd9c6a2d520f450674911bc7c13140f11b1aa41f8213b5feac8e355a78bc35d"} Aug 04 19:19:58.612249 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:58.611803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:19:58.629322 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:19:58.629277 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" podStartSLOduration=1.429787323 podStartE2EDuration="10.629262759s" totalImagesPullingTime="9.199475436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:19:58.627214378 +0000 UTC m=+259.597240900" watchObservedRunningTime="2026-08-04 19:19:58.629262759 +0000 UTC m=+259.599289277" Aug 04 19:20:02.109777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.109696 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84"] Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.109980 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="pull" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.109990 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="pull" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.110000 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="extract" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.110006 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="extract" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.110018 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="util" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.110024 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="util" Aug 04 19:20:02.110175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.110107 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb689ac7-550e-4e9e-8ba5-022d04ba0aaf" containerName="extract" Aug 04 19:20:02.151220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.151197 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84"] Aug 04 19:20:02.151358 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.151284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.154195 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.154173 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 19:20:02.154894 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.154877 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ggmgq\"" Aug 04 19:20:02.154982 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.154880 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:20:02.310094 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.310063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.310242 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.310108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8st64\" (UniqueName: \"kubernetes.io/projected/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-kube-api-access-8st64\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.410877 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.410796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8st64\" (UniqueName: \"kubernetes.io/projected/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-kube-api-access-8st64\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.411030 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.410884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.411260 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.411241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.420512 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.420479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8st64\" (UniqueName: \"kubernetes.io/projected/f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5-kube-api-access-8st64\") pod \"openshift-lws-operator-fd8ccff4c-g6l84\" (UID: \"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.459683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.459657 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" Aug 04 19:20:02.602060 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.602014 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84"] Aug 04 19:20:02.605242 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:02.605211 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b9986e_22fe_4e4f_bb52_98dd4f1a1cc5.slice/crio-e5e1af155d6b8f2df6a2cd3eb3ed1225313f6bf1c555e92d5198367f2b235dcf WatchSource:0}: Error finding container e5e1af155d6b8f2df6a2cd3eb3ed1225313f6bf1c555e92d5198367f2b235dcf: Status 404 returned error can't find the container with id e5e1af155d6b8f2df6a2cd3eb3ed1225313f6bf1c555e92d5198367f2b235dcf Aug 04 19:20:02.628013 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:02.627984 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" event={"ID":"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5","Type":"ContainerStarted","Data":"e5e1af155d6b8f2df6a2cd3eb3ed1225313f6bf1c555e92d5198367f2b235dcf"} Aug 04 19:20:04.617322 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:04.617296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vp2zc" Aug 04 19:20:05.641733 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:05.641693 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" event={"ID":"f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5","Type":"ContainerStarted","Data":"10d4fd42f2bcf2b8b8dddb085f87907a4d1efc074af3ca2217335dc69fdce9ad"} Aug 04 19:20:05.665843 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:05.665792 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6l84" podStartSLOduration=1.338064868 podStartE2EDuration="3.665773588s" totalImagesPullingTime="2.32770872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:05.66458052 +0000 UTC m=+266.634607040" watchObservedRunningTime="2026-08-04 19:20:05.665773588 +0000 UTC m=+266.635800109" Aug 04 19:20:08.120896 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.120867 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5"] Aug 04 19:20:08.123195 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.123179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.126239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.126214 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f9hlb\"" Aug 04 19:20:08.126346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.126249 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 19:20:08.127137 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.127119 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 19:20:08.133389 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.133370 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5"] Aug 04 19:20:08.254455 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.254428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.254569 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.254485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqjw\" (UniqueName: \"kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.254609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.254563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.355813 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.355786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqjw\" (UniqueName: \"kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.355914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.355842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.355914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.355871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.356240 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.356223 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.356315 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.356299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.365214 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.365196 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqjw\" (UniqueName: \"kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.432289 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.432237 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:08.554805 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.554783 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5"] Aug 04 19:20:08.556554 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:08.556529 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16be5a66_0a6a_4a23_8a73_4785ad7d7338.slice/crio-69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d WatchSource:0}: Error finding container 69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d: Status 404 returned error can't find the container with id 69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d Aug 04 19:20:08.678909 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.678876 2567 generic.go:356] "Generic (PLEG): container finished" podID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerID="2b91f19b4e544f2b7db92e0b6a67b6fbce732785200e006bd28c27332a331c78" exitCode=0 Aug 04 19:20:08.679061 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.678958 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" event={"ID":"16be5a66-0a6a-4a23-8a73-4785ad7d7338","Type":"ContainerDied","Data":"2b91f19b4e544f2b7db92e0b6a67b6fbce732785200e006bd28c27332a331c78"} Aug 04 19:20:08.679134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:08.679067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" event={"ID":"16be5a66-0a6a-4a23-8a73-4785ad7d7338","Type":"ContainerStarted","Data":"69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d"} Aug 04 19:20:09.687297 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:09.687222 2567 generic.go:356] "Generic (PLEG): container finished" podID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerID="71737a6b3553dbb73055f7c2693d6b73d85d91707899bd9aec8c16f7f6a91e73" exitCode=0 Aug 04 19:20:09.687297 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:09.687279 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" event={"ID":"16be5a66-0a6a-4a23-8a73-4785ad7d7338","Type":"ContainerDied","Data":"71737a6b3553dbb73055f7c2693d6b73d85d91707899bd9aec8c16f7f6a91e73"} Aug 04 19:20:10.693554 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:10.693517 2567 generic.go:356] "Generic (PLEG): container finished" podID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerID="fff5346af105cc64f307c6f13268508331618d4d21c659320821d1bf06fbd204" exitCode=0 Aug 04 19:20:10.693945 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:10.693597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" event={"ID":"16be5a66-0a6a-4a23-8a73-4785ad7d7338","Type":"ContainerDied","Data":"fff5346af105cc64f307c6f13268508331618d4d21c659320821d1bf06fbd204"} Aug 04 19:20:11.815855 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.815833 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:11.981440 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.981375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqjw\" (UniqueName: \"kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw\") pod \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " Aug 04 19:20:11.981440 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.981438 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle\") pod \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " Aug 04 19:20:11.981585 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.981490 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util\") pod \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\" (UID: \"16be5a66-0a6a-4a23-8a73-4785ad7d7338\") " Aug 04 19:20:11.982207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.982174 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle" (OuterVolumeSpecName: "bundle") pod "16be5a66-0a6a-4a23-8a73-4785ad7d7338" (UID: "16be5a66-0a6a-4a23-8a73-4785ad7d7338"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:20:11.983430 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.983404 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw" (OuterVolumeSpecName: "kube-api-access-7wqjw") pod "16be5a66-0a6a-4a23-8a73-4785ad7d7338" (UID: "16be5a66-0a6a-4a23-8a73-4785ad7d7338"). InnerVolumeSpecName "kube-api-access-7wqjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:20:11.987220 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:11.987099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util" (OuterVolumeSpecName: "util") pod "16be5a66-0a6a-4a23-8a73-4785ad7d7338" (UID: "16be5a66-0a6a-4a23-8a73-4785ad7d7338"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:20:12.082369 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.082344 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:20:12.082369 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.082366 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqjw\" (UniqueName: \"kubernetes.io/projected/16be5a66-0a6a-4a23-8a73-4785ad7d7338-kube-api-access-7wqjw\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:20:12.082498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.082377 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16be5a66-0a6a-4a23-8a73-4785ad7d7338-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:20:12.703907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.703874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" event={"ID":"16be5a66-0a6a-4a23-8a73-4785ad7d7338","Type":"ContainerDied","Data":"69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d"} Aug 04 19:20:12.703907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.703907 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm5kp5" Aug 04 19:20:12.704109 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:12.703920 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c4599eba0fdd34ccee093f23e35736d03c98a101b4db8a48f2c36ee113424d" Aug 04 19:20:22.180865 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.180831 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c"] Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181124 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="extract" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181137 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="extract" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181151 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="util" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181156 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="util" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181167 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="pull" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181172 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="pull" Aug 04 19:20:22.181281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.181223 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="16be5a66-0a6a-4a23-8a73-4785ad7d7338" containerName="extract" Aug 04 19:20:22.183968 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.183951 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.186528 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.186503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 19:20:22.186650 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.186555 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 19:20:22.186650 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.186616 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 19:20:22.186650 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.186560 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 19:20:22.186813 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.186747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9bp4b\"" Aug 04 19:20:22.200270 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.200251 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c"] Aug 04 19:20:22.352955 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.352917 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.352955 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.352961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6m7r\" (UniqueName: \"kubernetes.io/projected/1d52f445-4f93-4c04-8de7-e44602ed39fe-kube-api-access-c6m7r\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.353207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.353079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.353207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.353131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.353207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.353165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.453710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.453651 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.453710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.453686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.453710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.453705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.453918 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.453729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.453918 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.453754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6m7r\" (UniqueName: \"kubernetes.io/projected/1d52f445-4f93-4c04-8de7-e44602ed39fe-kube-api-access-c6m7r\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.454230 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.454206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.455991 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.455973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d52f445-4f93-4c04-8de7-e44602ed39fe-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.456229 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.456208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.456296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.456208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1d52f445-4f93-4c04-8de7-e44602ed39fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.462408 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.462382 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6m7r\" (UniqueName: \"kubernetes.io/projected/1d52f445-4f93-4c04-8de7-e44602ed39fe-kube-api-access-c6m7r\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-k7h2c\" (UID: \"1d52f445-4f93-4c04-8de7-e44602ed39fe\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.494092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.494071 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:22.628870 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.628848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c"] Aug 04 19:20:22.631246 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:22.631215 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d52f445_4f93_4c04_8de7_e44602ed39fe.slice/crio-de4e6b7c2ad589e6b2f43c1fd26080886e08c7086763bd77c8f373478b821d11 WatchSource:0}: Error finding container de4e6b7c2ad589e6b2f43c1fd26080886e08c7086763bd77c8f373478b821d11: Status 404 returned error can't find the container with id de4e6b7c2ad589e6b2f43c1fd26080886e08c7086763bd77c8f373478b821d11 Aug 04 19:20:22.745879 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:22.745814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" event={"ID":"1d52f445-4f93-4c04-8de7-e44602ed39fe","Type":"ContainerStarted","Data":"de4e6b7c2ad589e6b2f43c1fd26080886e08c7086763bd77c8f373478b821d11"} Aug 04 19:20:26.768301 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:26.768263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" event={"ID":"1d52f445-4f93-4c04-8de7-e44602ed39fe","Type":"ContainerStarted","Data":"79e27efb044097582fa928c6a207a6f159e5d725a5e66dc95c5b47ebfdac5eb4"} Aug 04 19:20:27.774285 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:27.774244 2567 generic.go:356] "Generic (PLEG): container finished" podID="1d52f445-4f93-4c04-8de7-e44602ed39fe" containerID="79e27efb044097582fa928c6a207a6f159e5d725a5e66dc95c5b47ebfdac5eb4" exitCode=0 Aug 04 19:20:27.774637 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:27.774292 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" event={"ID":"1d52f445-4f93-4c04-8de7-e44602ed39fe","Type":"ContainerDied","Data":"79e27efb044097582fa928c6a207a6f159e5d725a5e66dc95c5b47ebfdac5eb4"} Aug 04 19:20:28.780947 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:28.780858 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" event={"ID":"1d52f445-4f93-4c04-8de7-e44602ed39fe","Type":"ContainerStarted","Data":"047d5dacfa835c1afa10d5f357fb5ce783ba2b9e226d4e4de4a423358a049a29"} Aug 04 19:20:28.781330 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:28.780944 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:28.806245 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:28.806201 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" podStartSLOduration=2.151221523 podStartE2EDuration="6.806187321s" totalImagesPullingTime="3.59094566s" totalInitContainerRuntime="1.064020138s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:28.80349659 +0000 UTC m=+289.773523108" watchObservedRunningTime="2026-08-04 19:20:28.806187321 +0000 UTC m=+289.776213905" Aug 04 19:20:39.405734 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:39.405713 2567 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 19:20:39.787421 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:39.787399 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-k7h2c" Aug 04 19:20:42.377782 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.377747 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b452j"] Aug 04 19:20:42.380631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.380607 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.385909 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.385891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7nc2c\"" Aug 04 19:20:42.386887 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.386859 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 19:20:42.396698 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.396673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b452j"] Aug 04 19:20:42.502756 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.502725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe59b4a-e165-4a39-a277-b74d18a9e17d-cert\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.502907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.502761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp55q\" (UniqueName: \"kubernetes.io/projected/ebe59b4a-e165-4a39-a277-b74d18a9e17d-kube-api-access-lp55q\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.603451 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.603419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe59b4a-e165-4a39-a277-b74d18a9e17d-cert\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.603451 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.603452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp55q\" (UniqueName: \"kubernetes.io/projected/ebe59b4a-e165-4a39-a277-b74d18a9e17d-kube-api-access-lp55q\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.605818 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.605797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe59b4a-e165-4a39-a277-b74d18a9e17d-cert\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.616173 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.616152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp55q\" (UniqueName: \"kubernetes.io/projected/ebe59b4a-e165-4a39-a277-b74d18a9e17d-kube-api-access-lp55q\") pod \"odh-model-controller-7998c67dc4-b452j\" (UID: \"ebe59b4a-e165-4a39-a277-b74d18a9e17d\") " pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.697125 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.697072 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:42.826138 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.826113 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b452j"] Aug 04 19:20:42.828043 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:42.828020 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe59b4a_e165_4a39_a277_b74d18a9e17d.slice/crio-9f41b1f491a516a6deb1f506d2a2363a0a959f280314bdbae55f65fd98f7af42 WatchSource:0}: Error finding container 9f41b1f491a516a6deb1f506d2a2363a0a959f280314bdbae55f65fd98f7af42: Status 404 returned error can't find the container with id 9f41b1f491a516a6deb1f506d2a2363a0a959f280314bdbae55f65fd98f7af42 Aug 04 19:20:42.829553 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.829538 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:20:42.842620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:42.842587 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" event={"ID":"ebe59b4a-e165-4a39-a277-b74d18a9e17d","Type":"ContainerStarted","Data":"9f41b1f491a516a6deb1f506d2a2363a0a959f280314bdbae55f65fd98f7af42"} Aug 04 19:20:45.859570 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:45.859523 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" event={"ID":"ebe59b4a-e165-4a39-a277-b74d18a9e17d","Type":"ContainerStarted","Data":"75df27b7f60d6ab184dc32b037b85d21315b7dee90221943a1a9a0f180e06df5"} Aug 04 19:20:45.859896 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:45.859600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:45.889225 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:45.889181 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" podStartSLOduration=0.939487935 podStartE2EDuration="3.889166431s" totalImagesPullingTime="2.949678496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:45.888090854 +0000 UTC m=+306.858117383" watchObservedRunningTime="2026-08-04 19:20:45.889166431 +0000 UTC m=+306.859192949" Aug 04 19:20:47.783937 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.783908 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm"] Aug 04 19:20:47.786359 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.786341 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.789189 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.789169 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 19:20:47.789189 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.789181 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 19:20:47.789337 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.789170 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 19:20:47.789459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.789441 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 19:20:47.790253 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.790235 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2l9m2\"" Aug 04 19:20:47.811625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.811603 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm"] Aug 04 19:20:47.842550 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.842511 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczvf\" (UniqueName: \"kubernetes.io/projected/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-kube-api-access-vczvf\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.842652 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.842554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.842652 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.842588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.943451 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.943423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczvf\" (UniqueName: \"kubernetes.io/projected/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-kube-api-access-vczvf\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.943567 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.943459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.943567 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.943487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.945670 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.945627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tmp\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.945997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.945978 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-tls-certs\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:47.973347 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:47.973326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczvf\" (UniqueName: \"kubernetes.io/projected/c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29-kube-api-access-vczvf\") pod \"kube-auth-proxy-7fcb8f55f4-bqdrm\" (UID: \"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29\") " pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:48.096361 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:48.096297 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" Aug 04 19:20:48.236759 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:48.236729 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm"] Aug 04 19:20:48.239868 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:48.239843 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bae692_36bf_4d6f_8fb2_46a1eb4a2f29.slice/crio-d2cc72bc991437076c47ad2d8ca72d77b0ef5876de8f815e3c858a9b637b5a84 WatchSource:0}: Error finding container d2cc72bc991437076c47ad2d8ca72d77b0ef5876de8f815e3c858a9b637b5a84: Status 404 returned error can't find the container with id d2cc72bc991437076c47ad2d8ca72d77b0ef5876de8f815e3c858a9b637b5a84 Aug 04 19:20:48.873903 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:48.873861 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" event={"ID":"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29","Type":"ContainerStarted","Data":"d2cc72bc991437076c47ad2d8ca72d77b0ef5876de8f815e3c858a9b637b5a84"} Aug 04 19:20:50.084427 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.084390 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn"] Aug 04 19:20:50.094100 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.094078 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.099207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.099185 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 19:20:50.099657 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.099630 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-t52bz\"" Aug 04 19:20:50.099736 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.099677 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 19:20:50.099736 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.099726 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 19:20:50.100107 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.100090 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 19:20:50.114725 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.114672 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn"] Aug 04 19:20:50.164499 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.164776 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.164944 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.164944 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ntd\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-kube-api-access-58ntd\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.164944 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.165161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.164969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.165161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.165003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0cde63-27c5-4218-90f9-777a375b8855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266171 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ntd\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-kube-api-access-58ntd\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.266345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.266332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0cde63-27c5-4218-90f9-777a375b8855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.267150 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.267126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.268770 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.268730 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.268936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.268907 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0cde63-27c5-4218-90f9-777a375b8855-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.269020 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.268943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.269162 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.269147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0cde63-27c5-4218-90f9-777a375b8855-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.280301 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.280262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.281700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.281682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ntd\" (UniqueName: \"kubernetes.io/projected/ea0cde63-27c5-4218-90f9-777a375b8855-kube-api-access-58ntd\") pod \"istiod-openshift-gateway-5677ccb49b-wrhcn\" (UID: \"ea0cde63-27c5-4218-90f9-777a375b8855\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.405595 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.405541 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:50.564346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:50.564319 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn"] Aug 04 19:20:51.132118 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:51.132080 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0cde63_27c5_4218_90f9_777a375b8855.slice/crio-c29ce33ad4d354a119a082a6dad1df606b0a7752f10e688aca3211e089859551 WatchSource:0}: Error finding container c29ce33ad4d354a119a082a6dad1df606b0a7752f10e688aca3211e089859551: Status 404 returned error can't find the container with id c29ce33ad4d354a119a082a6dad1df606b0a7752f10e688aca3211e089859551 Aug 04 19:20:51.889847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:51.889798 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" event={"ID":"c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29","Type":"ContainerStarted","Data":"9f8b2f576343cc83790f025711b9d96b7c1a79c4970eb8ebe45bb88587b8c65d"} Aug 04 19:20:51.890892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:51.890869 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" event={"ID":"ea0cde63-27c5-4218-90f9-777a375b8855","Type":"ContainerStarted","Data":"c29ce33ad4d354a119a082a6dad1df606b0a7752f10e688aca3211e089859551"} Aug 04 19:20:54.248764 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.248700 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fcb8f55f4-bqdrm" podStartSLOduration=4.305937242 podStartE2EDuration="7.248678262s" totalImagesPullingTime="2.94274102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:51.972906093 +0000 UTC m=+312.942932613" watchObservedRunningTime="2026-08-04 19:20:54.248678262 +0000 UTC m=+315.218704784" Aug 04 19:20:54.250168 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.250145 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz"] Aug 04 19:20:54.253550 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.253533 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5"] Aug 04 19:20:54.253658 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.253640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.256638 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.256617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.260895 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.260875 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 04 19:20:54.261172 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.261151 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8w7rf\"" Aug 04 19:20:54.266698 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.266681 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-92ndv\"" Aug 04 19:20:54.289965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.286685 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 19:20:54.289965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.289573 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5"] Aug 04 19:20:54.292541 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.292520 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz"] Aug 04 19:20:54.303609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.303586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfc8k\" (UniqueName: \"kubernetes.io/projected/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-kube-api-access-rfc8k\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.303712 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.303632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdk6\" (UniqueName: \"kubernetes.io/projected/8980ee69-e170-494f-a6d8-87601698a3e2-kube-api-access-csdk6\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.303712 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.303670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8980ee69-e170-494f-a6d8-87601698a3e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.303816 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.303798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.404757 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.404719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.404929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.404832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfc8k\" (UniqueName: \"kubernetes.io/projected/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-kube-api-access-rfc8k\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.404929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.404904 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdk6\" (UniqueName: \"kubernetes.io/projected/8980ee69-e170-494f-a6d8-87601698a3e2-kube-api-access-csdk6\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.405063 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.404931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8980ee69-e170-494f-a6d8-87601698a3e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.408295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.408267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.408401 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.408338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8980ee69-e170-494f-a6d8-87601698a3e2-cert\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.422357 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.422334 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdk6\" (UniqueName: \"kubernetes.io/projected/8980ee69-e170-494f-a6d8-87601698a3e2-kube-api-access-csdk6\") pod \"kserve-controller-manager-d6f4b65b8-nr7x5\" (UID: \"8980ee69-e170-494f-a6d8-87601698a3e2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:54.428241 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.428217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfc8k\" (UniqueName: \"kubernetes.io/projected/6f9a58f8-acec-42f4-88c0-0d5a4c93e488-kube-api-access-rfc8k\") pod \"llmisvc-controller-manager-777c6bd7d5-6zqfz\" (UID: \"6f9a58f8-acec-42f4-88c0-0d5a4c93e488\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.566438 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.566393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:54.572402 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:54.572169 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:55.462238 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.461750 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:20:55.462238 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.461853 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:20:55.543544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.543518 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5"] Aug 04 19:20:55.544596 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:55.544574 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8980ee69_e170_494f_a6d8_87601698a3e2.slice/crio-1d718d00f24efd1e3712851ad8aebd21248724392e5b3fdf5043448d94a8ba5b WatchSource:0}: Error finding container 1d718d00f24efd1e3712851ad8aebd21248724392e5b3fdf5043448d94a8ba5b: Status 404 returned error can't find the container with id 1d718d00f24efd1e3712851ad8aebd21248724392e5b3fdf5043448d94a8ba5b Aug 04 19:20:55.577293 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.577255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz"] Aug 04 19:20:55.578526 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:20:55.578502 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f9a58f8_acec_42f4_88c0_0d5a4c93e488.slice/crio-666e9896c5342287928860ed5d37cbb2ca54a3f0d1a66905c6623299c5d3b0c6 WatchSource:0}: Error finding container 666e9896c5342287928860ed5d37cbb2ca54a3f0d1a66905c6623299c5d3b0c6: Status 404 returned error can't find the container with id 666e9896c5342287928860ed5d37cbb2ca54a3f0d1a66905c6623299c5d3b0c6 Aug 04 19:20:55.915572 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.915540 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" event={"ID":"ea0cde63-27c5-4218-90f9-777a375b8855","Type":"ContainerStarted","Data":"2aa49cba81d746544843b45f061225f1f58b6c6f230cbf043b13a10cd4f0c2fe"} Aug 04 19:20:55.915767 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.915656 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:55.916671 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.916642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" event={"ID":"6f9a58f8-acec-42f4-88c0-0d5a4c93e488","Type":"ContainerStarted","Data":"666e9896c5342287928860ed5d37cbb2ca54a3f0d1a66905c6623299c5d3b0c6"} Aug 04 19:20:55.917665 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.917647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" event={"ID":"8980ee69-e170-494f-a6d8-87601698a3e2","Type":"ContainerStarted","Data":"1d718d00f24efd1e3712851ad8aebd21248724392e5b3fdf5043448d94a8ba5b"} Aug 04 19:20:55.970397 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:55.970354 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" podStartSLOduration=1.642908227 podStartE2EDuration="5.970341739s" totalImagesPullingTime="4.327433512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:55.96948428 +0000 UTC m=+316.939510799" watchObservedRunningTime="2026-08-04 19:20:55.970341739 +0000 UTC m=+316.940368258" Aug 04 19:20:56.866366 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:56.866331 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-b452j" Aug 04 19:20:56.924225 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:56.924193 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-wrhcn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:20:56.924386 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:56.924253 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" podUID="ea0cde63-27c5-4218-90f9-777a375b8855" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:20:58.940069 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:58.940007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" event={"ID":"8980ee69-e170-494f-a6d8-87601698a3e2","Type":"ContainerStarted","Data":"3ff2e1c16ffb7ce60080a4f11cd00d32c5be58fa983db433205aaa9ae67ce198"} Aug 04 19:20:58.940530 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:58.940098 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:20:58.966092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:58.966022 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" podStartSLOduration=2.470842596 podStartE2EDuration="4.96600133s" totalImagesPullingTime="2.495158734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:58.963696573 +0000 UTC m=+319.933723111" watchObservedRunningTime="2026-08-04 19:20:58.96600133 +0000 UTC m=+319.936027850" Aug 04 19:20:59.923062 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:59.923033 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-wrhcn" Aug 04 19:20:59.945932 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:59.945894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" event={"ID":"6f9a58f8-acec-42f4-88c0-0d5a4c93e488","Type":"ContainerStarted","Data":"e4b13bf1f05217b15d8271b26d48df24d3b8d1bbfc586e326cae277a30f56c05"} Aug 04 19:20:59.946277 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:59.946089 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:20:59.978855 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:20:59.978806 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" podStartSLOduration=2.424985052 podStartE2EDuration="5.978791169s" totalImagesPullingTime="3.553806117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:59.977181482 +0000 UTC m=+320.947208002" watchObservedRunningTime="2026-08-04 19:20:59.978791169 +0000 UTC m=+320.948817687" Aug 04 19:21:29.951380 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:29.951353 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nr7x5" Aug 04 19:21:30.953249 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:30.953217 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zqfz" Aug 04 19:21:47.514421 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.514387 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz"] Aug 04 19:21:47.517937 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.517919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.523507 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.523482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 19:21:47.523653 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.523505 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f9hlb\"" Aug 04 19:21:47.523653 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.523482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 19:21:47.529246 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.529226 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz"] Aug 04 19:21:47.614101 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.614071 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht"] Aug 04 19:21:47.616614 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.616599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.622394 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.622367 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.622527 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.622408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vck6\" (UniqueName: \"kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.622527 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.622481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.632208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.632186 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht"] Aug 04 19:21:47.722916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.722883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.723105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.722937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.723105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.722960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.723105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.723028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphq2\" (UniqueName: \"kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.723105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.723099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.723348 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.723119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vck6\" (UniqueName: \"kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.723407 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.723363 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.723407 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.723396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.728777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.728754 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9"] Aug 04 19:21:47.731814 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.731793 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.733377 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.733356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vck6\" (UniqueName: \"kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.745352 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.745326 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9"] Aug 04 19:21:47.823796 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.823960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphq2\" (UniqueName: \"kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.823960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823829 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.823960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823887 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.823960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.823960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.823925 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4rh\" (UniqueName: \"kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.824266 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.824245 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.824340 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.824320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.827761 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.827742 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:47.832900 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.832878 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9"] Aug 04 19:21:47.835327 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.835306 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:47.841721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.841698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphq2\" (UniqueName: \"kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.851631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.851599 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9"] Aug 04 19:21:47.924311 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbv2\" (UniqueName: \"kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:47.924466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4rh\" (UniqueName: \"kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.924466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.924466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.924632 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924529 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:47.924632 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:47.924747 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.924796 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.925008 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.924994 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:47.935321 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.935292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4rh\" (UniqueName: \"kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:47.968481 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:47.968449 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz"] Aug 04 19:21:47.971661 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:21:47.971638 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5236923_6c65_48a4_9b26_95fd6486b9f4.slice/crio-30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a WatchSource:0}: Error finding container 30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a: Status 404 returned error can't find the container with id 30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a Aug 04 19:21:48.026154 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.026122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.026278 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.026179 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbv2\" (UniqueName: \"kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.026345 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.026303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.026610 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.026584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.026704 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.026651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.036639 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.036617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbv2\" (UniqueName: \"kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.048681 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.048656 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:48.070708 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.070682 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht"] Aug 04 19:21:48.072095 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:21:48.072072 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ff87ed_5c7d_452d_b830_4ee68e852e15.slice/crio-bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434 WatchSource:0}: Error finding container bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434: Status 404 returned error can't find the container with id bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434 Aug 04 19:21:48.162195 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.162172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:48.184181 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.184149 2567 generic.go:356] "Generic (PLEG): container finished" podID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerID="fb75c93372cf22e7a122fd352eae08097ab9bbcc9e3d78d249cc986646fba0e4" exitCode=0 Aug 04 19:21:48.184309 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.184228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" event={"ID":"f5236923-6c65-48a4-9b26-95fd6486b9f4","Type":"ContainerDied","Data":"fb75c93372cf22e7a122fd352eae08097ab9bbcc9e3d78d249cc986646fba0e4"} Aug 04 19:21:48.184309 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.184293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" event={"ID":"f5236923-6c65-48a4-9b26-95fd6486b9f4","Type":"ContainerStarted","Data":"30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a"} Aug 04 19:21:48.189995 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.189970 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9"] Aug 04 19:21:48.190232 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.190209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerStarted","Data":"3841a5099d61e46aa4babe3d503ec992deea1e91eb40f4c452978652da9455ee"} Aug 04 19:21:48.190332 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.190254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerStarted","Data":"bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434"} Aug 04 19:21:48.192140 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:21:48.192114 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5600c20c_1ae0_48ec_9423_7229f50a8427.slice/crio-b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca WatchSource:0}: Error finding container b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca: Status 404 returned error can't find the container with id b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca Aug 04 19:21:48.300456 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:48.300434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9"] Aug 04 19:21:48.301803 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:21:48.301782 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43b0516_64e0_4c9a_aa78_c280697245e8.slice/crio-88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43 WatchSource:0}: Error finding container 88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43: Status 404 returned error can't find the container with id 88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43 Aug 04 19:21:49.197586 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.197548 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerID="3841a5099d61e46aa4babe3d503ec992deea1e91eb40f4c452978652da9455ee" exitCode=0 Aug 04 19:21:49.198074 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.197625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerDied","Data":"3841a5099d61e46aa4babe3d503ec992deea1e91eb40f4c452978652da9455ee"} Aug 04 19:21:49.199549 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.199518 2567 generic.go:356] "Generic (PLEG): container finished" podID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerID="cf3550ce06de29f377b6db3fc96c22e9b19a2cb68978522bf4f8d97e72c99389" exitCode=0 Aug 04 19:21:49.199643 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.199594 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" event={"ID":"f5236923-6c65-48a4-9b26-95fd6486b9f4","Type":"ContainerDied","Data":"cf3550ce06de29f377b6db3fc96c22e9b19a2cb68978522bf4f8d97e72c99389"} Aug 04 19:21:49.201242 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.201215 2567 generic.go:356] "Generic (PLEG): container finished" podID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerID="47f991ae25fb3efa7dc14f40086bf84550515d3070b45b8a05113aae50edd5d3" exitCode=0 Aug 04 19:21:49.201337 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.201309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerDied","Data":"47f991ae25fb3efa7dc14f40086bf84550515d3070b45b8a05113aae50edd5d3"} Aug 04 19:21:49.201400 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.201349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerStarted","Data":"b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca"} Aug 04 19:21:49.202882 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.202859 2567 generic.go:356] "Generic (PLEG): container finished" podID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerID="01d60b1004922076cc8362830f5dae9c4cf779fb8571dae6a554a5c3edd36b52" exitCode=0 Aug 04 19:21:49.202962 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.202932 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerDied","Data":"01d60b1004922076cc8362830f5dae9c4cf779fb8571dae6a554a5c3edd36b52"} Aug 04 19:21:49.203005 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:49.202975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerStarted","Data":"88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43"} Aug 04 19:21:50.214131 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:50.214103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerStarted","Data":"e34e8645b417bffeefef232e4a9462d3aaa227ec715843441c35a7ea07ef6a1c"} Aug 04 19:21:50.216621 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:50.216594 2567 generic.go:356] "Generic (PLEG): container finished" podID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerID="274db0d4456ae7e122cb02e8ea89982a2cd0b34cc18d472c938f84d01de65cb4" exitCode=0 Aug 04 19:21:50.216729 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:50.216677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" event={"ID":"f5236923-6c65-48a4-9b26-95fd6486b9f4","Type":"ContainerDied","Data":"274db0d4456ae7e122cb02e8ea89982a2cd0b34cc18d472c938f84d01de65cb4"} Aug 04 19:21:50.218867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:50.218766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerStarted","Data":"1bb0f74da7292e5064755785d0e37ede852bfd6246727d28ab5493aaace4ae0b"} Aug 04 19:21:50.220964 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:50.220944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerStarted","Data":"e32627b43abda365946d96431444c08b94a9e0a325f89421b806687bcc5acd8f"} Aug 04 19:21:50.373210 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:21:50.373147 2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43b0516_64e0_4c9a_aa78_c280697245e8.slice/crio-e32627b43abda365946d96431444c08b94a9e0a325f89421b806687bcc5acd8f.scope\": RecentStats: unable to find data in memory cache]" Aug 04 19:21:51.228154 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.228108 2567 generic.go:356] "Generic (PLEG): container finished" podID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerID="1bb0f74da7292e5064755785d0e37ede852bfd6246727d28ab5493aaace4ae0b" exitCode=0 Aug 04 19:21:51.228563 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.228151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerDied","Data":"1bb0f74da7292e5064755785d0e37ede852bfd6246727d28ab5493aaace4ae0b"} Aug 04 19:21:51.230071 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.230038 2567 generic.go:356] "Generic (PLEG): container finished" podID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerID="e32627b43abda365946d96431444c08b94a9e0a325f89421b806687bcc5acd8f" exitCode=0 Aug 04 19:21:51.230161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.230080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerDied","Data":"e32627b43abda365946d96431444c08b94a9e0a325f89421b806687bcc5acd8f"} Aug 04 19:21:51.232182 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.232159 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerID="e34e8645b417bffeefef232e4a9462d3aaa227ec715843441c35a7ea07ef6a1c" exitCode=0 Aug 04 19:21:51.232326 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.232247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerDied","Data":"e34e8645b417bffeefef232e4a9462d3aaa227ec715843441c35a7ea07ef6a1c"} Aug 04 19:21:51.376159 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.376042 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:51.453957 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.453934 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vck6\" (UniqueName: \"kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6\") pod \"f5236923-6c65-48a4-9b26-95fd6486b9f4\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " Aug 04 19:21:51.454072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.453972 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util\") pod \"f5236923-6c65-48a4-9b26-95fd6486b9f4\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " Aug 04 19:21:51.454072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.454033 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle\") pod \"f5236923-6c65-48a4-9b26-95fd6486b9f4\" (UID: \"f5236923-6c65-48a4-9b26-95fd6486b9f4\") " Aug 04 19:21:51.454626 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.454595 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle" (OuterVolumeSpecName: "bundle") pod "f5236923-6c65-48a4-9b26-95fd6486b9f4" (UID: "f5236923-6c65-48a4-9b26-95fd6486b9f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:51.455892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.455862 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6" (OuterVolumeSpecName: "kube-api-access-8vck6") pod "f5236923-6c65-48a4-9b26-95fd6486b9f4" (UID: "f5236923-6c65-48a4-9b26-95fd6486b9f4"). InnerVolumeSpecName "kube-api-access-8vck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:21:51.459796 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.459771 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util" (OuterVolumeSpecName: "util") pod "f5236923-6c65-48a4-9b26-95fd6486b9f4" (UID: "f5236923-6c65-48a4-9b26-95fd6486b9f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:51.555241 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.555215 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vck6\" (UniqueName: \"kubernetes.io/projected/f5236923-6c65-48a4-9b26-95fd6486b9f4-kube-api-access-8vck6\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:51.555344 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.555243 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:51.555344 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:51.555257 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5236923-6c65-48a4-9b26-95fd6486b9f4-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:52.239876 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.239846 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerID="369ca748a49af5a75c20ed76e05129c466db163fd5a335ffa1aca043d1c056f6" exitCode=0 Aug 04 19:21:52.240306 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.239936 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerDied","Data":"369ca748a49af5a75c20ed76e05129c466db163fd5a335ffa1aca043d1c056f6"} Aug 04 19:21:52.241670 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.241654 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" Aug 04 19:21:52.241670 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.241662 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cz5sbz" event={"ID":"f5236923-6c65-48a4-9b26-95fd6486b9f4","Type":"ContainerDied","Data":"30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a"} Aug 04 19:21:52.241790 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.241696 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30478eaf36b0810a7e4c9de54e2c3224633be125390374b5256ea0653dfafb5a" Aug 04 19:21:52.244016 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.243995 2567 generic.go:356] "Generic (PLEG): container finished" podID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerID="e5e56d86d9d4962fbb91e8ac6787458dc654cf68640577bb3058fad7d58957e1" exitCode=0 Aug 04 19:21:52.244150 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.244035 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerDied","Data":"e5e56d86d9d4962fbb91e8ac6787458dc654cf68640577bb3058fad7d58957e1"} Aug 04 19:21:52.246117 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.246097 2567 generic.go:356] "Generic (PLEG): container finished" podID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerID="53e78a9f03fc39fbe9145781cfa59590c51d99fe3e21f94f8713edffd2d80c30" exitCode=0 Aug 04 19:21:52.246224 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:52.246155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerDied","Data":"53e78a9f03fc39fbe9145781cfa59590c51d99fe3e21f94f8713edffd2d80c30"} Aug 04 19:21:53.392453 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.392430 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:53.436623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.436600 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:21:53.440625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.440606 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:53.472779 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472757 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util\") pod \"5600c20c-1ae0-48ec-9423-7229f50a8427\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " Aug 04 19:21:53.472911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472786 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util\") pod \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " Aug 04 19:21:53.472911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472835 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle\") pod \"f43b0516-64e0-4c9a-aa78-c280697245e8\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " Aug 04 19:21:53.472911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472864 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dphq2\" (UniqueName: \"kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2\") pod \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " Aug 04 19:21:53.472911 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472889 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util\") pod \"f43b0516-64e0-4c9a-aa78-c280697245e8\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " Aug 04 19:21:53.473139 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.472923 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle\") pod \"5600c20c-1ae0-48ec-9423-7229f50a8427\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " Aug 04 19:21:53.473846 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.473816 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle" (OuterVolumeSpecName: "bundle") pod "f43b0516-64e0-4c9a-aa78-c280697245e8" (UID: "f43b0516-64e0-4c9a-aa78-c280697245e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.473954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.473877 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle\") pod \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\" (UID: \"e7ff87ed-5c7d-452d-b830-4ee68e852e15\") " Aug 04 19:21:53.473954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.473909 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhbv2\" (UniqueName: \"kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2\") pod \"f43b0516-64e0-4c9a-aa78-c280697245e8\" (UID: \"f43b0516-64e0-4c9a-aa78-c280697245e8\") " Aug 04 19:21:53.474085 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.473958 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r4rh\" (UniqueName: \"kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh\") pod \"5600c20c-1ae0-48ec-9423-7229f50a8427\" (UID: \"5600c20c-1ae0-48ec-9423-7229f50a8427\") " Aug 04 19:21:53.474289 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.474271 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.474934 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.474851 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle" (OuterVolumeSpecName: "bundle") pod "e7ff87ed-5c7d-452d-b830-4ee68e852e15" (UID: "e7ff87ed-5c7d-452d-b830-4ee68e852e15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.477498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.477260 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2" (OuterVolumeSpecName: "kube-api-access-dphq2") pod "e7ff87ed-5c7d-452d-b830-4ee68e852e15" (UID: "e7ff87ed-5c7d-452d-b830-4ee68e852e15"). InnerVolumeSpecName "kube-api-access-dphq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:21:53.477698 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.477609 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2" (OuterVolumeSpecName: "kube-api-access-rhbv2") pod "f43b0516-64e0-4c9a-aa78-c280697245e8" (UID: "f43b0516-64e0-4c9a-aa78-c280697245e8"). InnerVolumeSpecName "kube-api-access-rhbv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:21:53.478526 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.478499 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle" (OuterVolumeSpecName: "bundle") pod "5600c20c-1ae0-48ec-9423-7229f50a8427" (UID: "5600c20c-1ae0-48ec-9423-7229f50a8427"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.479661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.479615 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh" (OuterVolumeSpecName: "kube-api-access-2r4rh") pod "5600c20c-1ae0-48ec-9423-7229f50a8427" (UID: "5600c20c-1ae0-48ec-9423-7229f50a8427"). InnerVolumeSpecName "kube-api-access-2r4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:21:53.481934 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.481914 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util" (OuterVolumeSpecName: "util") pod "e7ff87ed-5c7d-452d-b830-4ee68e852e15" (UID: "e7ff87ed-5c7d-452d-b830-4ee68e852e15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.482022 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.481991 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util" (OuterVolumeSpecName: "util") pod "f43b0516-64e0-4c9a-aa78-c280697245e8" (UID: "f43b0516-64e0-4c9a-aa78-c280697245e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.483789 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.483753 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util" (OuterVolumeSpecName: "util") pod "5600c20c-1ae0-48ec-9423-7229f50a8427" (UID: "5600c20c-1ae0-48ec-9423-7229f50a8427"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:21:53.575160 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575135 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4rh\" (UniqueName: \"kubernetes.io/projected/5600c20c-1ae0-48ec-9423-7229f50a8427-kube-api-access-2r4rh\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575160 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575160 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575169 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575179 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphq2\" (UniqueName: \"kubernetes.io/projected/e7ff87ed-5c7d-452d-b830-4ee68e852e15-kube-api-access-dphq2\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575187 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f43b0516-64e0-4c9a-aa78-c280697245e8-util\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575195 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600c20c-1ae0-48ec-9423-7229f50a8427-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575203 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ff87ed-5c7d-452d-b830-4ee68e852e15-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:53.575312 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:53.575211 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbv2\" (UniqueName: \"kubernetes.io/projected/f43b0516-64e0-4c9a-aa78-c280697245e8-kube-api-access-rhbv2\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:21:54.257366 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.257340 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" Aug 04 19:21:54.257528 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.257373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09v8ht" event={"ID":"e7ff87ed-5c7d-452d-b830-4ee68e852e15","Type":"ContainerDied","Data":"bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434"} Aug 04 19:21:54.257528 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.257421 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbcd3666ae0ae30c1ffb3d2ada9165398619736fe68ee52f84565d20176ea434" Aug 04 19:21:54.259377 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.259357 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" Aug 04 19:21:54.259494 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.259360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7plc9" event={"ID":"5600c20c-1ae0-48ec-9423-7229f50a8427","Type":"ContainerDied","Data":"b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca"} Aug 04 19:21:54.259552 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.259491 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02543ecaf83f92ac9bbef501d4fccf5006037fa8d4010d50d659eaa859fc3ca" Aug 04 19:21:54.261486 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.261463 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" event={"ID":"f43b0516-64e0-4c9a-aa78-c280697245e8","Type":"ContainerDied","Data":"88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43"} Aug 04 19:21:54.261577 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.261503 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88822ace4b5b6c0f6dc7350cd180f4752655605055596ce95f349b14e8212a43" Aug 04 19:21:54.261577 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:21:54.261505 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053br9g9" Aug 04 19:22:00.794685 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.794649 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr"] Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.794981 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.794992 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795000 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795006 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795012 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="extract" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795018 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="extract" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795029 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795034 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="pull" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795040 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="extract" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795044 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="extract" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795065 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="util" Aug 04 19:22:00.795044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795070 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795078 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795087 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795096 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="pull" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795102 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="pull" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795112 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795117 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795122 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795127 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795133 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795138 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795146 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795151 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="util" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795199 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5600c20c-1ae0-48ec-9423-7229f50a8427" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795206 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43b0516-64e0-4c9a-aa78-c280697245e8" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795216 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ff87ed-5c7d-452d-b830-4ee68e852e15" containerName="extract" Aug 04 19:22:00.795432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.795222 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5236923-6c65-48a4-9b26-95fd6486b9f4" containerName="extract" Aug 04 19:22:00.799363 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.799345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:00.804192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.804171 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 19:22:00.804192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.804187 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dd9fb\"" Aug 04 19:22:00.804378 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.804192 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 19:22:00.817110 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.817088 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr"] Aug 04 19:22:00.935925 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:00.935897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjtk\" (UniqueName: \"kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk\") pod \"limitador-operator-controller-manager-6c549b6d85-pvxkr\" (UID: \"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:01.036556 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:01.036523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjtk\" (UniqueName: \"kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk\") pod \"limitador-operator-controller-manager-6c549b6d85-pvxkr\" (UID: \"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:01.047855 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:01.047795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjtk\" (UniqueName: \"kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk\") pod \"limitador-operator-controller-manager-6c549b6d85-pvxkr\" (UID: \"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:01.108666 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:01.108637 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:01.240796 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:01.240772 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr"] Aug 04 19:22:01.243172 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:22:01.243146 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6032aa7_9b54_4dbc_a27b_b94cb21cb645.slice/crio-b530853f4f4c9df3bfaca43d077c5df18a31ec318e45007218cd373bfe31796c WatchSource:0}: Error finding container b530853f4f4c9df3bfaca43d077c5df18a31ec318e45007218cd373bfe31796c: Status 404 returned error can't find the container with id b530853f4f4c9df3bfaca43d077c5df18a31ec318e45007218cd373bfe31796c Aug 04 19:22:01.297130 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:01.297098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" event={"ID":"c6032aa7-9b54-4dbc-a27b-b94cb21cb645","Type":"ContainerStarted","Data":"b530853f4f4c9df3bfaca43d077c5df18a31ec318e45007218cd373bfe31796c"} Aug 04 19:22:04.320295 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:04.320265 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" event={"ID":"c6032aa7-9b54-4dbc-a27b-b94cb21cb645","Type":"ContainerStarted","Data":"5d45671cad7692820dc926f109fcc9b4875126a909c7bec62df16773c620f629"} Aug 04 19:22:04.320640 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:04.320360 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:04.337605 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:04.337554 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" podStartSLOduration=1.5775640850000001 podStartE2EDuration="4.337536559s" totalImagesPullingTime="2.759972474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:04.336741148 +0000 UTC m=+385.306767666" watchObservedRunningTime="2026-08-04 19:22:04.337536559 +0000 UTC m=+385.307563079" Aug 04 19:22:10.274102 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.274070 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bpj2c"] Aug 04 19:22:10.277362 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.277340 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:10.280136 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.280119 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cngqm\"" Aug 04 19:22:10.293788 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.293760 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bpj2c"] Aug 04 19:22:10.418835 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.418803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p66f\" (UniqueName: \"kubernetes.io/projected/37cca5ab-b6db-4c04-9f43-b91d328aa13b-kube-api-access-4p66f\") pod \"authorino-operator-68d88948d8-bpj2c\" (UID: \"37cca5ab-b6db-4c04-9f43-b91d328aa13b\") " pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:10.519274 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.519240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p66f\" (UniqueName: \"kubernetes.io/projected/37cca5ab-b6db-4c04-9f43-b91d328aa13b-kube-api-access-4p66f\") pod \"authorino-operator-68d88948d8-bpj2c\" (UID: \"37cca5ab-b6db-4c04-9f43-b91d328aa13b\") " pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:10.530610 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.530554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p66f\" (UniqueName: \"kubernetes.io/projected/37cca5ab-b6db-4c04-9f43-b91d328aa13b-kube-api-access-4p66f\") pod \"authorino-operator-68d88948d8-bpj2c\" (UID: \"37cca5ab-b6db-4c04-9f43-b91d328aa13b\") " pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:10.586720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.586694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:10.729696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:10.729669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-bpj2c"] Aug 04 19:22:10.731772 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:22:10.731744 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cca5ab_b6db_4c04_9f43_b91d328aa13b.slice/crio-003fa02a0f59b2f6f9231fc9a9e18d1406c6cf642404fffe74be1b9025d62014 WatchSource:0}: Error finding container 003fa02a0f59b2f6f9231fc9a9e18d1406c6cf642404fffe74be1b9025d62014: Status 404 returned error can't find the container with id 003fa02a0f59b2f6f9231fc9a9e18d1406c6cf642404fffe74be1b9025d62014 Aug 04 19:22:11.358968 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:11.358931 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" event={"ID":"37cca5ab-b6db-4c04-9f43-b91d328aa13b","Type":"ContainerStarted","Data":"003fa02a0f59b2f6f9231fc9a9e18d1406c6cf642404fffe74be1b9025d62014"} Aug 04 19:22:13.371398 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:13.371366 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" event={"ID":"37cca5ab-b6db-4c04-9f43-b91d328aa13b","Type":"ContainerStarted","Data":"f62d70e02594f5354f46c88e728227dafd42cf9bc86e371ce45d8658f1ceadac"} Aug 04 19:22:13.371846 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:13.371512 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:13.391907 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:13.391860 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" podStartSLOduration=0.897471255 podStartE2EDuration="3.391846341s" totalImagesPullingTime="2.494375086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:13.389198299 +0000 UTC m=+394.359224818" watchObservedRunningTime="2026-08-04 19:22:13.391846341 +0000 UTC m=+394.361872861" Aug 04 19:22:15.327439 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:15.327407 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:24.379278 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:24.379246 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-bpj2c" Aug 04 19:22:25.300374 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.300339 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr"] Aug 04 19:22:25.300582 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.300561 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerName="manager" containerID="cri-o://5d45671cad7692820dc926f109fcc9b4875126a909c7bec62df16773c620f629" gracePeriod=2 Aug 04 19:22:25.314409 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.314380 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr"] Aug 04 19:22:25.325104 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.325075 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": dial tcp 10.133.0.35:8081: connect: connection refused" Aug 04 19:22:25.325671 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:22:25.325647 2567 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-pvxkr\": pods \"limitador-operator-controller-manager-6c549b6d85-pvxkr\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:25.334182 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.334161 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg"] Aug 04 19:22:25.334495 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.334483 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerName="manager" Aug 04 19:22:25.334556 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.334497 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerName="manager" Aug 04 19:22:25.334592 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.334568 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerName="manager" Aug 04 19:22:25.337531 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.337514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:25.342171 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:22:25.342148 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-pvxkr\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:25.358645 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.358604 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg"] Aug 04 19:22:25.433383 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.433355 2567 generic.go:356] "Generic (PLEG): container finished" podID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" containerID="5d45671cad7692820dc926f109fcc9b4875126a909c7bec62df16773c620f629" exitCode=0 Aug 04 19:22:25.448777 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.448748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lln25\" (UniqueName: \"kubernetes.io/projected/f75d7b5a-0ca4-4064-b208-7927b09d0cac-kube-api-access-lln25\") pod \"limitador-operator-controller-manager-6c549b6d85-xzkpg\" (UID: \"f75d7b5a-0ca4-4064-b208-7927b09d0cac\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:25.549504 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.549477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lln25\" (UniqueName: \"kubernetes.io/projected/f75d7b5a-0ca4-4064-b208-7927b09d0cac-kube-api-access-lln25\") pod \"limitador-operator-controller-manager-6c549b6d85-xzkpg\" (UID: \"f75d7b5a-0ca4-4064-b208-7927b09d0cac\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:25.553114 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.553064 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:25.559115 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.559081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lln25\" (UniqueName: \"kubernetes.io/projected/f75d7b5a-0ca4-4064-b208-7927b09d0cac-kube-api-access-lln25\") pod \"limitador-operator-controller-manager-6c549b6d85-xzkpg\" (UID: \"f75d7b5a-0ca4-4064-b208-7927b09d0cac\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:25.650275 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.650244 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqjtk\" (UniqueName: \"kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk\") pod \"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\" (UID: \"c6032aa7-9b54-4dbc-a27b-b94cb21cb645\") " Aug 04 19:22:25.652145 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.652125 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk" (OuterVolumeSpecName: "kube-api-access-jqjtk") pod "c6032aa7-9b54-4dbc-a27b-b94cb21cb645" (UID: "c6032aa7-9b54-4dbc-a27b-b94cb21cb645"). InnerVolumeSpecName "kube-api-access-jqjtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:22:25.685165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.685141 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:25.751571 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.751542 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjtk\" (UniqueName: \"kubernetes.io/projected/c6032aa7-9b54-4dbc-a27b-b94cb21cb645-kube-api-access-jqjtk\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:22:25.816914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:25.816890 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg"] Aug 04 19:22:25.818263 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:22:25.818237 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75d7b5a_0ca4_4064_b208_7927b09d0cac.slice/crio-edde6c0751dd6dd622fa7a5e2a3ce90cea5315079b8e90488ce64ccd3af613ac WatchSource:0}: Error finding container edde6c0751dd6dd622fa7a5e2a3ce90cea5315079b8e90488ce64ccd3af613ac: Status 404 returned error can't find the container with id edde6c0751dd6dd622fa7a5e2a3ce90cea5315079b8e90488ce64ccd3af613ac Aug 04 19:22:26.440235 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.440200 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" event={"ID":"f75d7b5a-0ca4-4064-b208-7927b09d0cac","Type":"ContainerStarted","Data":"7d349c91fee1b9c133a731b6fad06757ab4e2f69fdc83f49601b68ed7113b94b"} Aug 04 19:22:26.440674 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.440249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" event={"ID":"f75d7b5a-0ca4-4064-b208-7927b09d0cac","Type":"ContainerStarted","Data":"edde6c0751dd6dd622fa7a5e2a3ce90cea5315079b8e90488ce64ccd3af613ac"} Aug 04 19:22:26.440674 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.440310 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:26.441664 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.441648 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-pvxkr" Aug 04 19:22:26.441740 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.441690 2567 scope.go:117] "RemoveContainer" containerID="5d45671cad7692820dc926f109fcc9b4875126a909c7bec62df16773c620f629" Aug 04 19:22:26.460509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:26.460469 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" podStartSLOduration=1.460455111 podStartE2EDuration="1.460455111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:26.457900585 +0000 UTC m=+407.427927117" watchObservedRunningTime="2026-08-04 19:22:26.460455111 +0000 UTC m=+407.430481629" Aug 04 19:22:27.504382 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:27.504354 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6032aa7-9b54-4dbc-a27b-b94cb21cb645" path="/var/lib/kubelet/pods/c6032aa7-9b54-4dbc-a27b-b94cb21cb645/volumes" Aug 04 19:22:37.449815 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:37.449785 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xzkpg" Aug 04 19:22:56.325689 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.325653 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:22:56.330956 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.330936 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.341119 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.341097 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:22:56.343180 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.343161 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 19:22:56.349535 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.349504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xpfq2\"" Aug 04 19:22:56.384329 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.384306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g24b\" (UniqueName: \"kubernetes.io/projected/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-kube-api-access-7g24b\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.384403 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.384342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-config-file\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.387384 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.387354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:22:56.485525 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.485501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g24b\" (UniqueName: \"kubernetes.io/projected/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-kube-api-access-7g24b\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.485661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.485543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-config-file\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.486172 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.486151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-config-file\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.500727 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.500705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g24b\" (UniqueName: \"kubernetes.io/projected/64d3ba82-bbdf-408a-a47c-8a730eb58b4f-kube-api-access-7g24b\") pod \"limitador-limitador-85d9db4d5d-njkqg\" (UID: \"64d3ba82-bbdf-408a-a47c-8a730eb58b4f\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.642866 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.642808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:56.775520 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:22:56.775492 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d3ba82_bbdf_408a_a47c_8a730eb58b4f.slice/crio-93a594bab7251eadf76e5907192a8b35141e76cc4403a73b882aef42d6f14dba WatchSource:0}: Error finding container 93a594bab7251eadf76e5907192a8b35141e76cc4403a73b882aef42d6f14dba: Status 404 returned error can't find the container with id 93a594bab7251eadf76e5907192a8b35141e76cc4403a73b882aef42d6f14dba Aug 04 19:22:56.781853 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:56.781829 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:22:57.599526 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:57.599486 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" event={"ID":"64d3ba82-bbdf-408a-a47c-8a730eb58b4f","Type":"ContainerStarted","Data":"93a594bab7251eadf76e5907192a8b35141e76cc4403a73b882aef42d6f14dba"} Aug 04 19:22:58.606572 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:58.606539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" event={"ID":"64d3ba82-bbdf-408a-a47c-8a730eb58b4f","Type":"ContainerStarted","Data":"fa3f1db7d63684799d0f2ed37f60b1de2bd87fd99377b23948e08fe7e8f01b48"} Aug 04 19:22:58.606922 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:58.606596 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:22:58.631421 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:22:58.631375 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" podStartSLOduration=1.734915591 podStartE2EDuration="2.631360439s" totalImagesPullingTime="896.444848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:58.629735457 +0000 UTC m=+439.599761978" watchObservedRunningTime="2026-08-04 19:22:58.631360439 +0000 UTC m=+439.601386974" Aug 04 19:23:09.612481 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:23:09.612456 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-njkqg" Aug 04 19:24:07.544991 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:24:07.544953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:24:25.820243 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:24:25.820203 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:24:33.096750 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:24:33.096672 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:25:00.694025 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:25:00.693991 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:25:25.899083 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:25:25.898101 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:25:34.411896 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:25:34.411860 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:25:49.594373 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:25:49.594337 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:27:12.307123 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:27:12.307084 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:27:22.691844 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:27:22.691805 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:27:35.710823 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:27:35.710757 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:29:15.944396 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:29:15.944359 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:29:25.396513 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:29:25.396473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:30:00.141925 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.141891 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq"] Aug 04 19:30:00.145210 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.145194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" Aug 04 19:30:00.147614 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.147596 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:30:00.152019 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.151998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq"] Aug 04 19:30:00.156246 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.156230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" Aug 04 19:30:00.285654 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.285626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq"] Aug 04 19:30:00.287140 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:30:00.287110 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79879ad_0384_49be_9171_b578704b9c53.slice/crio-22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8 WatchSource:0}: Error finding container 22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8: Status 404 returned error can't find the container with id 22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8 Aug 04 19:30:00.289351 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.289330 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:30:00.719466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:00.719426 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" event={"ID":"e79879ad-0384-49be-9171-b578704b9c53","Type":"ContainerStarted","Data":"22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8"} Aug 04 19:30:02.732952 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:02.732873 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" event={"ID":"e79879ad-0384-49be-9171-b578704b9c53","Type":"ContainerStarted","Data":"a4a9ee93e7ae96a46d93e5e11c335211d0e6e47c60ec4d17e63928ab536d0b7d"} Aug 04 19:30:02.749826 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:02.749785 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" podStartSLOduration=0.570586467 podStartE2EDuration="2.749769953s" totalImagesPullingTime="2.179183486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:30:02.747294607 +0000 UTC m=+863.717321127" watchObservedRunningTime="2026-08-04 19:30:02.749769953 +0000 UTC m=+863.719796516" Aug 04 19:30:03.739432 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:03.739398 2567 generic.go:356] "Generic (PLEG): container finished" podID="e79879ad-0384-49be-9171-b578704b9c53" containerID="a4a9ee93e7ae96a46d93e5e11c335211d0e6e47c60ec4d17e63928ab536d0b7d" exitCode=0 Aug 04 19:30:03.739790 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:03.739457 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" event={"ID":"e79879ad-0384-49be-9171-b578704b9c53","Type":"ContainerDied","Data":"a4a9ee93e7ae96a46d93e5e11c335211d0e6e47c60ec4d17e63928ab536d0b7d"} Aug 04 19:30:04.872785 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:04.872762 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" Aug 04 19:30:05.750700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:05.750673 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" Aug 04 19:30:05.750851 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:05.750667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq" event={"ID":"e79879ad-0384-49be-9171-b578704b9c53","Type":"ContainerDied","Data":"22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8"} Aug 04 19:30:05.750851 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:05.750802 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f7abe5c1670a44fe127493b285b1dd79335a721013ed17466f6b24a9feccb8" Aug 04 19:30:25.288032 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:25.287994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:30:32.084996 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:32.084923 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:30:47.492464 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:47.492435 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:30:54.489509 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:30:54.489473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:03.993620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:03.993584 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:13.294310 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:13.294260 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:25.883134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:25.883098 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:31.193061 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:31.193005 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:39.597105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:39.597045 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:48.695284 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:48.695247 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:31:56.575336 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.575299 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-695c89c559-kmlj2"] Aug 04 19:31:56.575724 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.575662 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79879ad-0384-49be-9171-b578704b9c53" containerName="cleanup" Aug 04 19:31:56.575724 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.575675 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79879ad-0384-49be-9171-b578704b9c53" containerName="cleanup" Aug 04 19:31:56.575795 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.575736 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79879ad-0384-49be-9171-b578704b9c53" containerName="cleanup" Aug 04 19:31:56.578682 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.578662 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.611764 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.611729 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695c89c559-kmlj2"] Aug 04 19:31:56.755745 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-console-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.755929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-service-ca\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.755929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-oauth-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.755929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2gh\" (UniqueName: \"kubernetes.io/projected/fe0b59e5-e704-43f1-97a2-13b888e71392-kube-api-access-ls2gh\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.755929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.755929 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-trusted-ca-bundle\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.756148 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.755946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-oauth-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857075 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-trusted-ca-bundle\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-oauth-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-console-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-service-ca\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-oauth-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857496 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2gh\" (UniqueName: \"kubernetes.io/projected/fe0b59e5-e704-43f1-97a2-13b888e71392-kube-api-access-ls2gh\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.857994 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.857973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-console-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.858136 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.858113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-trusted-ca-bundle\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.858208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.858111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-service-ca\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.858208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.858154 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0b59e5-e704-43f1-97a2-13b888e71392-oauth-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.859783 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.859761 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-serving-cert\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.859873 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.859855 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0b59e5-e704-43f1-97a2-13b888e71392-console-oauth-config\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.865105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.865081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2gh\" (UniqueName: \"kubernetes.io/projected/fe0b59e5-e704-43f1-97a2-13b888e71392-kube-api-access-ls2gh\") pod \"console-695c89c559-kmlj2\" (UID: \"fe0b59e5-e704-43f1-97a2-13b888e71392\") " pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:56.888070 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:56.888015 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:31:57.026979 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:57.026950 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695c89c559-kmlj2"] Aug 04 19:31:57.028692 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:31:57.028663 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0b59e5_e704_43f1_97a2_13b888e71392.slice/crio-3e81a7c6e49b46b83d5dfe1fc78604f3addd48c64ce1aac49d8de293cd5c940f WatchSource:0}: Error finding container 3e81a7c6e49b46b83d5dfe1fc78604f3addd48c64ce1aac49d8de293cd5c940f: Status 404 returned error can't find the container with id 3e81a7c6e49b46b83d5dfe1fc78604f3addd48c64ce1aac49d8de293cd5c940f Aug 04 19:31:57.326754 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:57.326720 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695c89c559-kmlj2" event={"ID":"fe0b59e5-e704-43f1-97a2-13b888e71392","Type":"ContainerStarted","Data":"a3ac031df50fc6b9efff515fa2b1e5299406bb19a6f83160d5119480421811ed"} Aug 04 19:31:57.326921 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:57.326766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695c89c559-kmlj2" event={"ID":"fe0b59e5-e704-43f1-97a2-13b888e71392","Type":"ContainerStarted","Data":"3e81a7c6e49b46b83d5dfe1fc78604f3addd48c64ce1aac49d8de293cd5c940f"} Aug 04 19:31:57.358990 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:31:57.358933 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-695c89c559-kmlj2" podStartSLOduration=1.358917189 podStartE2EDuration="1.358917189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:31:57.358291185 +0000 UTC m=+978.328317705" watchObservedRunningTime="2026-08-04 19:31:57.358917189 +0000 UTC m=+978.328943704" Aug 04 19:32:01.000600 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:01.000569 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:32:06.889028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:06.888938 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:32:06.889478 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:06.889295 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:32:06.893840 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:06.893822 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:32:07.381705 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:07.381679 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695c89c559-kmlj2" Aug 04 19:32:07.430059 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:07.430020 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:32:11.590986 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:11.590955 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:32:32.450248 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.450184 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59bcc48bdd-5xwrd" podUID="0858f19c-9f97-4d00-ba11-ded087643873" containerName="console" containerID="cri-o://47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca" gracePeriod=15 Aug 04 19:32:32.699273 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.699247 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:32:32.764868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.764803 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmsgt\" (UniqueName: \"kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.764868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.764840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.764874 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.764915 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.764943 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765072 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765064 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765272 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765097 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca\") pod \"0858f19c-9f97-4d00-ba11-ded087643873\" (UID: \"0858f19c-9f97-4d00-ba11-ded087643873\") " Aug 04 19:32:32.765359 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765323 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:32:32.765359 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765321 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config" (OuterVolumeSpecName: "console-config") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:32:32.765560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:32:32.765560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.765537 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca" (OuterVolumeSpecName: "service-ca") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:32:32.766920 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.766895 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt" (OuterVolumeSpecName: "kube-api-access-dmsgt") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "kube-api-access-dmsgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:32:32.767411 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.767387 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:32:32.767477 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.767413 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0858f19c-9f97-4d00-ba11-ded087643873" (UID: "0858f19c-9f97-4d00-ba11-ded087643873"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:32:32.866506 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866481 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866506 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866504 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866513 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsgt\" (UniqueName: \"kubernetes.io/projected/0858f19c-9f97-4d00-ba11-ded087643873-kube-api-access-dmsgt\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866523 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-trusted-ca-bundle\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866531 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0858f19c-9f97-4d00-ba11-ded087643873-console-oauth-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866539 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-console-config\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:32.866644 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:32.866547 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0858f19c-9f97-4d00-ba11-ded087643873-oauth-serving-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:32:33.511506 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.511466 2567 generic.go:356] "Generic (PLEG): container finished" podID="0858f19c-9f97-4d00-ba11-ded087643873" containerID="47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca" exitCode=0 Aug 04 19:32:33.511980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.511512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bcc48bdd-5xwrd" Aug 04 19:32:33.511980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.511558 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bcc48bdd-5xwrd" event={"ID":"0858f19c-9f97-4d00-ba11-ded087643873","Type":"ContainerDied","Data":"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca"} Aug 04 19:32:33.511980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.511615 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bcc48bdd-5xwrd" event={"ID":"0858f19c-9f97-4d00-ba11-ded087643873","Type":"ContainerDied","Data":"afedbdac8651a5405244bc2dcf830dfdbc33b527132005f249a5296c3e86fa4c"} Aug 04 19:32:33.511980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.511642 2567 scope.go:117] "RemoveContainer" containerID="47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca" Aug 04 19:32:33.522553 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.522538 2567 scope.go:117] "RemoveContainer" containerID="47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca" Aug 04 19:32:33.522805 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:32:33.522785 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca\": container with ID starting with 47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca not found: ID does not exist" containerID="47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca" Aug 04 19:32:33.522868 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.522812 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca"} err="failed to get container status \"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca\": rpc error: code = NotFound desc = could not find container \"47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca\": container with ID starting with 47a6700c60c43631ef025770b2e25520ba0ca5ff40fcacbf177a14a31a0824ca not found: ID does not exist" Aug 04 19:32:33.528200 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.528177 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:32:33.534291 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:33.534271 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59bcc48bdd-5xwrd"] Aug 04 19:32:34.395466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:34.395432 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:32:35.506234 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:35.506202 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0858f19c-9f97-4d00-ba11-ded087643873" path="/var/lib/kubelet/pods/0858f19c-9f97-4d00-ba11-ded087643873/volumes" Aug 04 19:32:44.793770 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:44.793733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:32:50.001341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:50.001308 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:32:59.095417 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:32:59.095384 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:10.290240 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:10.290203 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:16.293127 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:16.293096 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:23.998305 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:23.998270 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:37.995417 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:37.995331 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:43.492194 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:43.492153 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:53.693940 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:53.693910 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:33:58.788245 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:33:58.788209 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:34:08.889795 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:34:08.889756 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:34:14.186490 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:34:14.186456 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:34:20.297799 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:34:20.297766 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:34:29.889623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:34:29.889592 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:36:44.690442 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:36:44.690360 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:36:50.099548 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:36:50.099508 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:37:15.793977 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:37:15.793937 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:37:50.690086 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:37:50.690040 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:37:59.590651 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:37:59.590623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:06.288734 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:06.288657 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:12.304667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:12.304634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:21.193234 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:21.193199 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:29.492332 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:29.492302 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:37.094012 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:37.093976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:42.185810 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:42.185776 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:51.699942 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:51.699908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:38:59.092535 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:38:59.092497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:39:08.989232 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:39:08.989204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:39:34.303672 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:39:34.303595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:39:39.696469 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:39:39.696435 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:39:45.893209 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:39:45.893173 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:39:56.884706 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:39:56.884674 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:40:08.285038 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:40:08.285004 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:40:19.505322 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:40:19.505292 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:40:27.790511 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:40:27.790478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:40:39.986925 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:40:39.986882 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:40:55.794408 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:40:55.794369 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:00.185882 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:00.185850 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:10.491842 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:10.491813 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:15.888434 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:15.888402 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:33.692088 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:33.692044 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:39.584858 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:39.584824 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:41:48.591619 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:41:48.591578 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:00.984903 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:00.984863 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:10.191413 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:10.191377 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:17.686758 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:17.686724 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:25.893624 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:25.893591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:34.294830 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:34.294720 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:43.883916 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:43.883881 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:42:51.001799 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:42:51.001759 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:00.796084 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:00.796036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:16.306008 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:16.305974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:22.502086 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:22.502033 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:30.553177 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:30.553144 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:40.196416 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:40.193831 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:48.492029 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:48.491994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:43:54.294697 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:43:54.294663 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:44:04.399078 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.398978 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:04.399639 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.399603 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0858f19c-9f97-4d00-ba11-ded087643873" containerName="console" Aug 04 19:44:04.399639 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.399628 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0858f19c-9f97-4d00-ba11-ded087643873" containerName="console" Aug 04 19:44:04.399838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.399728 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0858f19c-9f97-4d00-ba11-ded087643873" containerName="console" Aug 04 19:44:04.402899 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.402881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.405718 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.405695 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:44:04.405835 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.405732 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-89a8e771-serving-cert\"" Aug 04 19:44:04.406740 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.406722 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:44:04.406846 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.406728 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:44:04.411960 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.411937 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:04.539689 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.539662 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qnd\" (UniqueName: \"kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.539828 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.539772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.539828 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.539805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.641281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.641233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.641437 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.641304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.641437 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.641369 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qnd\" (UniqueName: \"kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.641889 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.641863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.643716 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.643689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.650623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.650568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qnd\" (UniqueName: \"kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd\") pod \"maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.714750 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.714726 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:04.850773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.850746 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:04.852814 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:44:04.852785 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fa4d99_c8e7_407f_8cc0_797d5fd73728.slice/crio-fe61307ebb60e07d8959abc5b10f3ff183e8b8b394e97a2dadb236e5074c494f WatchSource:0}: Error finding container fe61307ebb60e07d8959abc5b10f3ff183e8b8b394e97a2dadb236e5074c494f: Status 404 returned error can't find the container with id fe61307ebb60e07d8959abc5b10f3ff183e8b8b394e97a2dadb236e5074c494f Aug 04 19:44:04.854514 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:04.854498 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:44:05.021631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:05.021555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" event={"ID":"25fa4d99-c8e7-407f-8cc0-797d5fd73728","Type":"ContainerStarted","Data":"fe61307ebb60e07d8959abc5b10f3ff183e8b8b394e97a2dadb236e5074c494f"} Aug 04 19:44:07.034297 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:07.034261 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" event={"ID":"25fa4d99-c8e7-407f-8cc0-797d5fd73728","Type":"ContainerStarted","Data":"541338c0cfaaa3289c951370c9466f9e5bd6140adff35acef632a5a2ed144fab"} Aug 04 19:44:07.034709 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:07.034324 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:07.050557 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:07.050498 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" podStartSLOduration=1.469824955 podStartE2EDuration="3.05047905s" totalImagesPullingTime="1.580654095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:07.050457118 +0000 UTC m=+1708.020483638" watchObservedRunningTime="2026-08-04 19:44:07.05047905 +0000 UTC m=+1708.020505570" Aug 04 19:44:07.191110 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:07.191081 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:44:13.046529 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:13.046503 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:28.865190 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:28.865154 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg"] Aug 04 19:44:28.868493 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:28.868477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:28.871036 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:28.871014 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:44:28.876127 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:28.875459 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg"] Aug 04 19:44:28.927112 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:28.927084 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg\" (UID: \"4993e9b7-adda-4282-bf41-0616e3e40562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:29.027780 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:29.027738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg\" (UID: \"4993e9b7-adda-4282-bf41-0616e3e40562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:29.028363 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:29.028346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg\" (UID: \"4993e9b7-adda-4282-bf41-0616e3e40562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:29.179651 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:29.179596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:29.510341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:29.510314 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg"] Aug 04 19:44:29.511182 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:44:29.511155 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4993e9b7_adda_4282_bf41_0616e3e40562.slice/crio-bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c WatchSource:0}: Error finding container bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c: Status 404 returned error can't find the container with id bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c Aug 04 19:44:30.153105 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:30.153000 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" event={"ID":"4993e9b7-adda-4282-bf41-0616e3e40562","Type":"ContainerStarted","Data":"be703aae262074505ba0500fd041707c96c4bd02a7d986a13a00db1cf165d60c"} Aug 04 19:44:30.153426 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:30.153116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" event={"ID":"4993e9b7-adda-4282-bf41-0616e3e40562","Type":"ContainerStarted","Data":"bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c"} Aug 04 19:44:30.170927 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:30.170881 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" podStartSLOduration=2.170867769 podStartE2EDuration="2.170867769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:30.168163748 +0000 UTC m=+1731.138190268" watchObservedRunningTime="2026-08-04 19:44:30.170867769 +0000 UTC m=+1731.140894286" Aug 04 19:44:31.159614 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:31.159581 2567 generic.go:356] "Generic (PLEG): container finished" podID="4993e9b7-adda-4282-bf41-0616e3e40562" containerID="be703aae262074505ba0500fd041707c96c4bd02a7d986a13a00db1cf165d60c" exitCode=0 Aug 04 19:44:31.160017 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:31.159664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" event={"ID":"4993e9b7-adda-4282-bf41-0616e3e40562","Type":"ContainerDied","Data":"be703aae262074505ba0500fd041707c96c4bd02a7d986a13a00db1cf165d60c"} Aug 04 19:44:32.290713 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:32.290694 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:32.351888 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:32.351863 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca\") pod \"4993e9b7-adda-4282-bf41-0616e3e40562\" (UID: \"4993e9b7-adda-4282-bf41-0616e3e40562\") " Aug 04 19:44:32.352207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:32.352185 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4993e9b7-adda-4282-bf41-0616e3e40562" (UID: "4993e9b7-adda-4282-bf41-0616e3e40562"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:44:32.453465 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:32.453401 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4993e9b7-adda-4282-bf41-0616e3e40562-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:44:33.171007 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:33.170980 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" Aug 04 19:44:33.171007 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:33.170995 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg" event={"ID":"4993e9b7-adda-4282-bf41-0616e3e40562","Type":"ContainerDied","Data":"bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c"} Aug 04 19:44:33.171212 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:33.171036 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5557888b9ce3a6e822dff375c985aa2c6249b5432bd79c01c77f54e4cc554c" Aug 04 19:44:35.046809 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.046769 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:35.047296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.046979 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" podUID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" containerName="maas-api" containerID="cri-o://541338c0cfaaa3289c951370c9466f9e5bd6140adff35acef632a5a2ed144fab" gracePeriod=30 Aug 04 19:44:35.182555 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.182519 2567 generic.go:356] "Generic (PLEG): container finished" podID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" containerID="541338c0cfaaa3289c951370c9466f9e5bd6140adff35acef632a5a2ed144fab" exitCode=0 Aug 04 19:44:35.182704 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.182593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" event={"ID":"25fa4d99-c8e7-407f-8cc0-797d5fd73728","Type":"ContainerDied","Data":"541338c0cfaaa3289c951370c9466f9e5bd6140adff35acef632a5a2ed144fab"} Aug 04 19:44:35.246445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.246411 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg"] Aug 04 19:44:35.249289 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.249259 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89a8e771-6gbxg"] Aug 04 19:44:35.296450 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.296424 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:35.376831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.376757 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca\") pod \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " Aug 04 19:44:35.376831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.376794 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls\") pod \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " Aug 04 19:44:35.376831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.376826 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qnd\" (UniqueName: \"kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd\") pod \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\" (UID: \"25fa4d99-c8e7-407f-8cc0-797d5fd73728\") " Aug 04 19:44:35.377186 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.377162 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25fa4d99-c8e7-407f-8cc0-797d5fd73728" (UID: "25fa4d99-c8e7-407f-8cc0-797d5fd73728"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:44:35.379765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.379741 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd" (OuterVolumeSpecName: "kube-api-access-b9qnd") pod "25fa4d99-c8e7-407f-8cc0-797d5fd73728" (UID: "25fa4d99-c8e7-407f-8cc0-797d5fd73728"). InnerVolumeSpecName "kube-api-access-b9qnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:44:35.379867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.379836 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25fa4d99-c8e7-407f-8cc0-797d5fd73728" (UID: "25fa4d99-c8e7-407f-8cc0-797d5fd73728"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:44:35.478477 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.478452 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fa4d99-c8e7-407f-8cc0-797d5fd73728-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.478477 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.478478 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fa4d99-c8e7-407f-8cc0-797d5fd73728-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.478628 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.478489 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qnd\" (UniqueName: \"kubernetes.io/projected/25fa4d99-c8e7-407f-8cc0-797d5fd73728-kube-api-access-b9qnd\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:44:35.503996 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:35.503971 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4993e9b7-adda-4282-bf41-0616e3e40562" path="/var/lib/kubelet/pods/4993e9b7-adda-4282-bf41-0616e3e40562/volumes" Aug 04 19:44:36.189018 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:36.188989 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" Aug 04 19:44:36.189401 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:36.188986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r" event={"ID":"25fa4d99-c8e7-407f-8cc0-797d5fd73728","Type":"ContainerDied","Data":"fe61307ebb60e07d8959abc5b10f3ff183e8b8b394e97a2dadb236e5074c494f"} Aug 04 19:44:36.190228 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:36.189157 2567 scope.go:117] "RemoveContainer" containerID="541338c0cfaaa3289c951370c9466f9e5bd6140adff35acef632a5a2ed144fab" Aug 04 19:44:36.210755 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:36.210724 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:36.214284 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:36.214264 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89a8e771-5fb44ccc58-6fl5r"] Aug 04 19:44:37.505186 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:37.505153 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" path="/var/lib/kubelet/pods/25fa4d99-c8e7-407f-8cc0-797d5fd73728/volumes" Aug 04 19:44:38.398597 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.398564 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:44:38.753959 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.753885 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754249 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4993e9b7-adda-4282-bf41-0616e3e40562" containerName="revoke-keys" Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754260 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4993e9b7-adda-4282-bf41-0616e3e40562" containerName="revoke-keys" Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754278 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" containerName="maas-api" Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754285 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" containerName="maas-api" Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754334 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4993e9b7-adda-4282-bf41-0616e3e40562" containerName="revoke-keys" Aug 04 19:44:38.754412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.754346 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fa4d99-c8e7-407f-8cc0-797d5fd73728" containerName="maas-api" Aug 04 19:44:38.758587 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.758567 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.761593 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.761574 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:44:38.761808 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.761793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ead82904-serving-cert\"" Aug 04 19:44:38.762711 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.762695 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:44:38.762776 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.762737 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:44:38.774706 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.774684 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:44:38.807175 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.807154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.807329 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.807200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bpr\" (UniqueName: \"kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.807329 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.807238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.907791 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.907759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.907913 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.907817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bpr\" (UniqueName: \"kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.907913 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.907858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.907986 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:44:38.907907 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-serving-cert: secret "maas-api-e2e-migrate-ead82904-serving-cert" not found Aug 04 19:44:38.907986 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:44:38.907980 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls podName:7861beb9-9d25-4d93-97ba-4815b56dd849 nodeName:}" failed. No retries permitted until 2026-08-04 19:44:39.407962034 +0000 UTC m=+1740.377988536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls") pod "maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" (UID: "7861beb9-9d25-4d93-97ba-4815b56dd849") : secret "maas-api-e2e-migrate-ead82904-serving-cert" not found Aug 04 19:44:38.908484 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.908469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:38.921109 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:38.921087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bpr\" (UniqueName: \"kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:39.412910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:39.412883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:39.415254 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:39.415226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") pod \"maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:39.673094 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:39.673021 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:44:39.678602 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:39.678583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:39.821532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:39.821509 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:44:39.823316 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:44:39.823289 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7861beb9_9d25_4d93_97ba_4815b56dd849.slice/crio-0b87bf4f78eb1f07fecd003644f5b16a212e8b71e4ddf674624582e13b8ce214 WatchSource:0}: Error finding container 0b87bf4f78eb1f07fecd003644f5b16a212e8b71e4ddf674624582e13b8ce214: Status 404 returned error can't find the container with id 0b87bf4f78eb1f07fecd003644f5b16a212e8b71e4ddf674624582e13b8ce214 Aug 04 19:44:40.211590 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:40.211554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" event={"ID":"7861beb9-9d25-4d93-97ba-4815b56dd849","Type":"ContainerStarted","Data":"0b87bf4f78eb1f07fecd003644f5b16a212e8b71e4ddf674624582e13b8ce214"} Aug 04 19:44:41.218624 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:41.218586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" event={"ID":"7861beb9-9d25-4d93-97ba-4815b56dd849","Type":"ContainerStarted","Data":"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea"} Aug 04 19:44:41.219025 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:41.218755 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:44:41.237489 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:41.237447 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" podStartSLOduration=2.714777919 podStartE2EDuration="3.237431447s" totalImagesPullingTime="522.653528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:41.23675084 +0000 UTC m=+1742.206777358" watchObservedRunningTime="2026-08-04 19:44:41.237431447 +0000 UTC m=+1742.207457968" Aug 04 19:44:45.397782 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:45.397750 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:44:47.230717 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:44:47.230690 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:45:00.150661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.150625 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk"] Aug 04 19:45:00.159595 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.159565 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88"] Aug 04 19:45:00.160195 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.160163 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" Aug 04 19:45:00.162970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.162948 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:45:00.163787 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.163530 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88"] Aug 04 19:45:00.163787 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.163623 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" Aug 04 19:45:00.165239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.165209 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk"] Aug 04 19:45:00.174924 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.174909 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" Aug 04 19:45:00.182182 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.182166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" Aug 04 19:45:00.336343 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.336316 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk"] Aug 04 19:45:00.337462 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:00.337436 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaee190c_6c13_4229_9728_02a2aa0bf325.slice/crio-d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115 WatchSource:0}: Error finding container d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115: Status 404 returned error can't find the container with id d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115 Aug 04 19:45:00.527006 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:00.526985 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88"] Aug 04 19:45:00.528667 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:00.528635 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590e6cca_e4c1_42ff_bb46_c53c4588fb6a.slice/crio-33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa WatchSource:0}: Error finding container 33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa: Status 404 returned error can't find the container with id 33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa Aug 04 19:45:01.334233 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.334197 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" event={"ID":"590e6cca-e4c1-42ff-bb46-c53c4588fb6a","Type":"ContainerStarted","Data":"836ac9c8534c1f58df12eb8cd3418223245f7973f2e120d4491766c5fa1a9b41"} Aug 04 19:45:01.334668 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.334255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" event={"ID":"590e6cca-e4c1-42ff-bb46-c53c4588fb6a","Type":"ContainerStarted","Data":"33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa"} Aug 04 19:45:01.335864 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.335831 2567 generic.go:356] "Generic (PLEG): container finished" podID="daee190c-6c13-4229-9728-02a2aa0bf325" containerID="0b65e66e5a4d1b4cb430522451a337a60cf0d82622f8ea1a6a75ff9e3cb45d41" exitCode=0 Aug 04 19:45:01.335969 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.335871 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" event={"ID":"daee190c-6c13-4229-9728-02a2aa0bf325","Type":"ContainerDied","Data":"0b65e66e5a4d1b4cb430522451a337a60cf0d82622f8ea1a6a75ff9e3cb45d41"} Aug 04 19:45:01.335969 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.335912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" event={"ID":"daee190c-6c13-4229-9728-02a2aa0bf325","Type":"ContainerStarted","Data":"d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115"} Aug 04 19:45:01.433722 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:01.433658 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" podStartSLOduration=1.433639175 podStartE2EDuration="1.433639175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:01.382477659 +0000 UTC m=+1762.352504179" watchObservedRunningTime="2026-08-04 19:45:01.433639175 +0000 UTC m=+1762.403665694" Aug 04 19:45:02.341908 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:02.341874 2567 generic.go:356] "Generic (PLEG): container finished" podID="590e6cca-e4c1-42ff-bb46-c53c4588fb6a" containerID="836ac9c8534c1f58df12eb8cd3418223245f7973f2e120d4491766c5fa1a9b41" exitCode=0 Aug 04 19:45:02.342314 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:02.341959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" event={"ID":"590e6cca-e4c1-42ff-bb46-c53c4588fb6a","Type":"ContainerDied","Data":"836ac9c8534c1f58df12eb8cd3418223245f7973f2e120d4491766c5fa1a9b41"} Aug 04 19:45:02.478214 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:02.478188 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" Aug 04 19:45:03.348152 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.348121 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" Aug 04 19:45:03.348152 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.348142 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764545-9fkwk" event={"ID":"daee190c-6c13-4229-9728-02a2aa0bf325","Type":"ContainerDied","Data":"d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115"} Aug 04 19:45:03.348622 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.348197 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e92b72f6a8f85b5e88559e04353d983e86fe81840506c79207cc1f96c6c115" Aug 04 19:45:03.478152 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.478131 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" Aug 04 19:45:03.628179 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628109 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56"] Aug 04 19:45:03.628448 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628437 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daee190c-6c13-4229-9728-02a2aa0bf325" containerName="cleanup" Aug 04 19:45:03.628490 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628450 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="daee190c-6c13-4229-9728-02a2aa0bf325" containerName="cleanup" Aug 04 19:45:03.628490 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628471 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590e6cca-e4c1-42ff-bb46-c53c4588fb6a" containerName="cleanup" Aug 04 19:45:03.628490 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628476 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="590e6cca-e4c1-42ff-bb46-c53c4588fb6a" containerName="cleanup" Aug 04 19:45:03.628589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628537 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="daee190c-6c13-4229-9728-02a2aa0bf325" containerName="cleanup" Aug 04 19:45:03.628589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.628545 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="590e6cca-e4c1-42ff-bb46-c53c4588fb6a" containerName="cleanup" Aug 04 19:45:03.631142 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.631128 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:03.643945 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.643920 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56"] Aug 04 19:45:03.703927 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.703891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ead82904-5bg56\" (UID: \"cd744196-d555-417e-826f-6cf0b46f84ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:03.804771 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.804738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ead82904-5bg56\" (UID: \"cd744196-d555-417e-826f-6cf0b46f84ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:03.805389 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.805369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ead82904-5bg56\" (UID: \"cd744196-d555-417e-826f-6cf0b46f84ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:03.941032 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:03.940949 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:04.073758 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.073733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56"] Aug 04 19:45:04.075251 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:04.075222 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd744196_d555_417e_826f_6cf0b46f84ef.slice/crio-1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62 WatchSource:0}: Error finding container 1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62: Status 404 returned error can't find the container with id 1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62 Aug 04 19:45:04.355008 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.354983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" event={"ID":"cd744196-d555-417e-826f-6cf0b46f84ef","Type":"ContainerStarted","Data":"bc0f92ba0a15ce37926a4d5b6dda4179729d25423d0049c459a39723814a4165"} Aug 04 19:45:04.355372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.355028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" event={"ID":"cd744196-d555-417e-826f-6cf0b46f84ef","Type":"ContainerStarted","Data":"1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62"} Aug 04 19:45:04.356219 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.356202 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" Aug 04 19:45:04.356276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.356208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88" event={"ID":"590e6cca-e4c1-42ff-bb46-c53c4588fb6a","Type":"ContainerDied","Data":"33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa"} Aug 04 19:45:04.356276 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.356248 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33918427a7ccb7bad6449d2daea4a26e6d7a35d97eca5205ebbf83c0094881fa" Aug 04 19:45:04.373979 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:04.373940 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" podStartSLOduration=1.373927914 podStartE2EDuration="1.373927914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:04.370464227 +0000 UTC m=+1765.340490745" watchObservedRunningTime="2026-08-04 19:45:04.373927914 +0000 UTC m=+1765.343954488" Aug 04 19:45:05.362750 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:05.362718 2567 generic.go:356] "Generic (PLEG): container finished" podID="cd744196-d555-417e-826f-6cf0b46f84ef" containerID="bc0f92ba0a15ce37926a4d5b6dda4179729d25423d0049c459a39723814a4165" exitCode=0 Aug 04 19:45:05.363170 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:05.362806 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" event={"ID":"cd744196-d555-417e-826f-6cf0b46f84ef","Type":"ContainerDied","Data":"bc0f92ba0a15ce37926a4d5b6dda4179729d25423d0049c459a39723814a4165"} Aug 04 19:45:06.496734 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:06.496711 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:06.631069 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:06.630968 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca\") pod \"cd744196-d555-417e-826f-6cf0b46f84ef\" (UID: \"cd744196-d555-417e-826f-6cf0b46f84ef\") " Aug 04 19:45:06.631336 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:06.631314 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd744196-d555-417e-826f-6cf0b46f84ef" (UID: "cd744196-d555-417e-826f-6cf0b46f84ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:06.732079 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:06.732038 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd744196-d555-417e-826f-6cf0b46f84ef-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:07.374817 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:07.374784 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" Aug 04 19:45:07.374974 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:07.374831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56" event={"ID":"cd744196-d555-417e-826f-6cf0b46f84ef","Type":"ContainerDied","Data":"1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62"} Aug 04 19:45:07.374974 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:07.374878 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1615ea84fc24ed315fc185ba3f07c824d2b05bb3d13206070859f1ced518de62" Aug 04 19:45:09.432769 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.432739 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:45:09.433245 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.432981 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" podUID="7861beb9-9d25-4d93-97ba-4815b56dd849" containerName="maas-api" containerID="cri-o://03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea" gracePeriod=30 Aug 04 19:45:09.458200 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.458176 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88"] Aug 04 19:45:09.462093 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.462072 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-ead82904-29764545-s2l88"] Aug 04 19:45:09.507143 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.507102 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590e6cca-e4c1-42ff-bb46-c53c4588fb6a" path="/var/lib/kubelet/pods/590e6cca-e4c1-42ff-bb46-c53c4588fb6a/volumes" Aug 04 19:45:09.672074 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.663504 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56"] Aug 04 19:45:09.675009 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.674973 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ead82904-5bg56"] Aug 04 19:45:09.678137 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.678119 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:45:09.860001 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.859970 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca\") pod \"7861beb9-9d25-4d93-97ba-4815b56dd849\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " Aug 04 19:45:09.860202 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.860039 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") pod \"7861beb9-9d25-4d93-97ba-4815b56dd849\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " Aug 04 19:45:09.860202 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.860088 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5bpr\" (UniqueName: \"kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr\") pod \"7861beb9-9d25-4d93-97ba-4815b56dd849\" (UID: \"7861beb9-9d25-4d93-97ba-4815b56dd849\") " Aug 04 19:45:09.860484 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.860457 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7861beb9-9d25-4d93-97ba-4815b56dd849" (UID: "7861beb9-9d25-4d93-97ba-4815b56dd849"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:09.862120 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.862097 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7861beb9-9d25-4d93-97ba-4815b56dd849" (UID: "7861beb9-9d25-4d93-97ba-4815b56dd849"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:45:09.862223 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.862202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr" (OuterVolumeSpecName: "kube-api-access-d5bpr") pod "7861beb9-9d25-4d93-97ba-4815b56dd849" (UID: "7861beb9-9d25-4d93-97ba-4815b56dd849"). InnerVolumeSpecName "kube-api-access-d5bpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:09.961459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.961422 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7861beb9-9d25-4d93-97ba-4815b56dd849-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:09.961459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.961452 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5bpr\" (UniqueName: \"kubernetes.io/projected/7861beb9-9d25-4d93-97ba-4815b56dd849-kube-api-access-d5bpr\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:09.961459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:09.961465 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7861beb9-9d25-4d93-97ba-4815b56dd849-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:10.391691 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.391658 2567 generic.go:356] "Generic (PLEG): container finished" podID="7861beb9-9d25-4d93-97ba-4815b56dd849" containerID="03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea" exitCode=0 Aug 04 19:45:10.391871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.391702 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" Aug 04 19:45:10.391871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.391746 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" event={"ID":"7861beb9-9d25-4d93-97ba-4815b56dd849","Type":"ContainerDied","Data":"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea"} Aug 04 19:45:10.391871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.391836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw" event={"ID":"7861beb9-9d25-4d93-97ba-4815b56dd849","Type":"ContainerDied","Data":"0b87bf4f78eb1f07fecd003644f5b16a212e8b71e4ddf674624582e13b8ce214"} Aug 04 19:45:10.391871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.391864 2567 scope.go:117] "RemoveContainer" containerID="03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea" Aug 04 19:45:10.401270 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.401250 2567 scope.go:117] "RemoveContainer" containerID="03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea" Aug 04 19:45:10.401507 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:45:10.401489 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea\": container with ID starting with 03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea not found: ID does not exist" containerID="03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea" Aug 04 19:45:10.401557 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.401514 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea"} err="failed to get container status \"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea\": rpc error: code = NotFound desc = could not find container \"03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea\": container with ID starting with 03c35aa69355f251f5f36fdcc0292dce97efeee8448e88dbfda1805b8aed0aea not found: ID does not exist" Aug 04 19:45:10.416596 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.416575 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:45:10.429165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:10.429145 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ead82904-68865dc7dd-k2ppw"] Aug 04 19:45:11.504782 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:11.504747 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7861beb9-9d25-4d93-97ba-4815b56dd849" path="/var/lib/kubelet/pods/7861beb9-9d25-4d93-97ba-4815b56dd849/volumes" Aug 04 19:45:11.505149 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:11.505080 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd744196-d555-417e-826f-6cf0b46f84ef" path="/var/lib/kubelet/pods/cd744196-d555-417e-826f-6cf0b46f84ef/volumes" Aug 04 19:45:12.720863 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:12.720828 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:45:13.797871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.797840 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:13.798372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798314 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd744196-d555-417e-826f-6cf0b46f84ef" containerName="revoke-keys" Aug 04 19:45:13.798372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798334 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd744196-d555-417e-826f-6cf0b46f84ef" containerName="revoke-keys" Aug 04 19:45:13.798372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798350 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7861beb9-9d25-4d93-97ba-4815b56dd849" containerName="maas-api" Aug 04 19:45:13.798372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798360 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7861beb9-9d25-4d93-97ba-4815b56dd849" containerName="maas-api" Aug 04 19:45:13.798599 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798464 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd744196-d555-417e-826f-6cf0b46f84ef" containerName="revoke-keys" Aug 04 19:45:13.798599 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.798478 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7861beb9-9d25-4d93-97ba-4815b56dd849" containerName="maas-api" Aug 04 19:45:13.801896 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.801879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:13.804218 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.804201 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:45:13.804360 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.804343 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:45:13.805217 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.805203 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c53c7015-serving-cert\"" Aug 04 19:45:13.805267 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.805218 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:45:13.810898 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.810873 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:13.996073 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.996028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:13.996211 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.996089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:13.996211 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:13.996185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zdm\" (UniqueName: \"kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.097208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.097144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdm\" (UniqueName: \"kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.097208 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.097192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.097377 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.097235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.097895 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.097877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.099558 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.099540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.105329 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.105307 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zdm\" (UniqueName: \"kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm\") pod \"maas-api-e2e-ait-c53c7015-7889c5447-2lztm\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.112961 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.112941 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:14.270632 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.270604 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:14.271995 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:14.271969 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea5776b_254a_4bba_87c9_0349827f10cb.slice/crio-1184bf9b4877c80c56dfa934ede4ecf7c3083f45705eee8ccb17ea2db1385242 WatchSource:0}: Error finding container 1184bf9b4877c80c56dfa934ede4ecf7c3083f45705eee8ccb17ea2db1385242: Status 404 returned error can't find the container with id 1184bf9b4877c80c56dfa934ede4ecf7c3083f45705eee8ccb17ea2db1385242 Aug 04 19:45:14.418790 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:14.418706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" event={"ID":"fea5776b-254a-4bba-87c9-0349827f10cb","Type":"ContainerStarted","Data":"1184bf9b4877c80c56dfa934ede4ecf7c3083f45705eee8ccb17ea2db1385242"} Aug 04 19:45:15.425247 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:15.425213 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" event={"ID":"fea5776b-254a-4bba-87c9-0349827f10cb","Type":"ContainerStarted","Data":"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4"} Aug 04 19:45:15.425650 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:15.425317 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:15.444862 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:15.444823 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" podStartSLOduration=2.085799775 podStartE2EDuration="2.444809405s" totalImagesPullingTime="359.00963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:15.442267245 +0000 UTC m=+1776.412293762" watchObservedRunningTime="2026-08-04 19:45:15.444809405 +0000 UTC m=+1776.414835925" Aug 04 19:45:21.435427 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:21.435399 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:22.609392 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:22.609354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:45:38.495443 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.495409 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4"] Aug 04 19:45:38.499059 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.499036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:38.501744 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.501727 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:45:38.507906 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.507881 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4"] Aug 04 19:45:38.584107 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.584081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4\" (UID: \"dd084358-54b9-406f-a3f6-f25b0a3af4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:38.684700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.684673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4\" (UID: \"dd084358-54b9-406f-a3f6-f25b0a3af4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:38.685324 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.685303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4\" (UID: \"dd084358-54b9-406f-a3f6-f25b0a3af4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:38.811521 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.811495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:38.942341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:38.942314 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4"] Aug 04 19:45:38.943872 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:38.943849 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd084358_54b9_406f_a3f6_f25b0a3af4dc.slice/crio-3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3 WatchSource:0}: Error finding container 3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3: Status 404 returned error can't find the container with id 3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3 Aug 04 19:45:39.551180 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:39.551150 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" event={"ID":"dd084358-54b9-406f-a3f6-f25b0a3af4dc","Type":"ContainerStarted","Data":"682e8f01179c82cdb5a11bf704c186270986dc789d7b97da0e540fb24d6f8d97"} Aug 04 19:45:39.551583 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:39.551210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" event={"ID":"dd084358-54b9-406f-a3f6-f25b0a3af4dc","Type":"ContainerStarted","Data":"3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3"} Aug 04 19:45:39.567667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:39.567625 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" podStartSLOduration=1.5676116329999998 podStartE2EDuration="1.567611633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:39.566132598 +0000 UTC m=+1800.536159139" watchObservedRunningTime="2026-08-04 19:45:39.567611633 +0000 UTC m=+1800.537638152" Aug 04 19:45:40.557958 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:40.557924 2567 generic.go:356] "Generic (PLEG): container finished" podID="dd084358-54b9-406f-a3f6-f25b0a3af4dc" containerID="682e8f01179c82cdb5a11bf704c186270986dc789d7b97da0e540fb24d6f8d97" exitCode=0 Aug 04 19:45:40.558365 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:40.557970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" event={"ID":"dd084358-54b9-406f-a3f6-f25b0a3af4dc","Type":"ContainerDied","Data":"682e8f01179c82cdb5a11bf704c186270986dc789d7b97da0e540fb24d6f8d97"} Aug 04 19:45:41.691300 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:41.691277 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:41.809590 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:41.809553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca\") pod \"dd084358-54b9-406f-a3f6-f25b0a3af4dc\" (UID: \"dd084358-54b9-406f-a3f6-f25b0a3af4dc\") " Aug 04 19:45:41.809888 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:41.809866 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd084358-54b9-406f-a3f6-f25b0a3af4dc" (UID: "dd084358-54b9-406f-a3f6-f25b0a3af4dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:41.910322 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:41.910250 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd084358-54b9-406f-a3f6-f25b0a3af4dc-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:42.568564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:42.568534 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" Aug 04 19:45:42.568564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:42.568550 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4" event={"ID":"dd084358-54b9-406f-a3f6-f25b0a3af4dc","Type":"ContainerDied","Data":"3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3"} Aug 04 19:45:42.568763 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:42.568599 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3298bc598c79967ea01d1e60b82ec565a9f4e38b03f1320f189c392ed9d0b2a3" Aug 04 19:45:44.305077 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.305007 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:44.305523 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.305269 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" podUID="fea5776b-254a-4bba-87c9-0349827f10cb" containerName="maas-api" containerID="cri-o://cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4" gracePeriod=30 Aug 04 19:45:44.504204 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.504174 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4"] Aug 04 19:45:44.506154 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.506131 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c53c7015-mlfj4"] Aug 04 19:45:44.556893 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.556835 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:44.582736 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.582705 2567 generic.go:356] "Generic (PLEG): container finished" podID="fea5776b-254a-4bba-87c9-0349827f10cb" containerID="cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4" exitCode=0 Aug 04 19:45:44.582875 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.582745 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" Aug 04 19:45:44.582875 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.582783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" event={"ID":"fea5776b-254a-4bba-87c9-0349827f10cb","Type":"ContainerDied","Data":"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4"} Aug 04 19:45:44.582875 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.582834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm" event={"ID":"fea5776b-254a-4bba-87c9-0349827f10cb","Type":"ContainerDied","Data":"1184bf9b4877c80c56dfa934ede4ecf7c3083f45705eee8ccb17ea2db1385242"} Aug 04 19:45:44.582875 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.582858 2567 scope.go:117] "RemoveContainer" containerID="cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4" Aug 04 19:45:44.591794 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.591775 2567 scope.go:117] "RemoveContainer" containerID="cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4" Aug 04 19:45:44.592138 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:45:44.592032 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4\": container with ID starting with cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4 not found: ID does not exist" containerID="cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4" Aug 04 19:45:44.592138 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.592086 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4"} err="failed to get container status \"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4\": rpc error: code = NotFound desc = could not find container \"cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4\": container with ID starting with cbed4e492c92b438eb29be2a36b80f975272b33ddf862c7c37d53b7ee24cbfa4 not found: ID does not exist" Aug 04 19:45:44.634158 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.634135 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9zdm\" (UniqueName: \"kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm\") pod \"fea5776b-254a-4bba-87c9-0349827f10cb\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " Aug 04 19:45:44.634257 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.634170 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls\") pod \"fea5776b-254a-4bba-87c9-0349827f10cb\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " Aug 04 19:45:44.634257 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.634193 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca\") pod \"fea5776b-254a-4bba-87c9-0349827f10cb\" (UID: \"fea5776b-254a-4bba-87c9-0349827f10cb\") " Aug 04 19:45:44.634628 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.634600 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fea5776b-254a-4bba-87c9-0349827f10cb" (UID: "fea5776b-254a-4bba-87c9-0349827f10cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:44.636738 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.636718 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fea5776b-254a-4bba-87c9-0349827f10cb" (UID: "fea5776b-254a-4bba-87c9-0349827f10cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:45:44.636738 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.636727 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm" (OuterVolumeSpecName: "kube-api-access-r9zdm") pod "fea5776b-254a-4bba-87c9-0349827f10cb" (UID: "fea5776b-254a-4bba-87c9-0349827f10cb"). InnerVolumeSpecName "kube-api-access-r9zdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:44.734998 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.734964 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zdm\" (UniqueName: \"kubernetes.io/projected/fea5776b-254a-4bba-87c9-0349827f10cb-kube-api-access-r9zdm\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.734998 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.734992 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fea5776b-254a-4bba-87c9-0349827f10cb-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.734998 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.735001 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fea5776b-254a-4bba-87c9-0349827f10cb-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:45:44.920779 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.920749 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:44.944239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:44.944203 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c53c7015-7889c5447-2lztm"] Aug 04 19:45:45.504404 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:45.504370 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd084358-54b9-406f-a3f6-f25b0a3af4dc" path="/var/lib/kubelet/pods/dd084358-54b9-406f-a3f6-f25b0a3af4dc/volumes" Aug 04 19:45:45.504790 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:45.504757 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fea5776b-254a-4bba-87c9-0349827f10cb" path="/var/lib/kubelet/pods/fea5776b-254a-4bba-87c9-0349827f10cb/volumes" Aug 04 19:45:45.696776 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:45.696745 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:45:50.060503 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.060466 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:45:50.060990 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.060969 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd084358-54b9-406f-a3f6-f25b0a3af4dc" containerName="revoke-keys" Aug 04 19:45:50.061092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.060994 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd084358-54b9-406f-a3f6-f25b0a3af4dc" containerName="revoke-keys" Aug 04 19:45:50.061092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.061019 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea5776b-254a-4bba-87c9-0349827f10cb" containerName="maas-api" Aug 04 19:45:50.061092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.061028 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea5776b-254a-4bba-87c9-0349827f10cb" containerName="maas-api" Aug 04 19:45:50.061249 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.061155 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea5776b-254a-4bba-87c9-0349827f10cb" containerName="maas-api" Aug 04 19:45:50.061249 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.061172 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd084358-54b9-406f-a3f6-f25b0a3af4dc" containerName="revoke-keys" Aug 04 19:45:50.065604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.065584 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.068625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.068597 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3d6cfa2c-serving-cert\"" Aug 04 19:45:50.068738 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.068601 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:45:50.069804 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.069788 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:45:50.069899 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.069789 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:45:50.075397 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.075366 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:45:50.186257 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.186219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hd5\" (UniqueName: \"kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.186508 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.186387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.186508 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.186444 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.287026 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.286992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.287199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.287035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.287199 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.287108 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hd5\" (UniqueName: \"kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.287275 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:45:50.287225 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-serving-cert: secret "maas-api-e2e-derive-3d6cfa2c-serving-cert" not found Aug 04 19:45:50.287314 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:45:50.287297 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls podName:34ec8e30-11bc-4059-a371-2023dcd0e08a nodeName:}" failed. No retries permitted until 2026-08-04 19:45:50.787280126 +0000 UTC m=+1811.757306627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls") pod "maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" (UID: "34ec8e30-11bc-4059-a371-2023dcd0e08a") : secret "maas-api-e2e-derive-3d6cfa2c-serving-cert" not found Aug 04 19:45:50.287693 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.287675 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.296297 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.296276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hd5\" (UniqueName: \"kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.790881 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.790846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.793255 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.793235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") pod \"maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:50.977040 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:50.977010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:51.125625 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:51.125590 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:45:51.126750 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:45:51.126725 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ec8e30_11bc_4059_a371_2023dcd0e08a.slice/crio-0bdccf2d314989ccb03561168011a03ad2efa08de15198d5b91c5dc46e9fff20 WatchSource:0}: Error finding container 0bdccf2d314989ccb03561168011a03ad2efa08de15198d5b91c5dc46e9fff20: Status 404 returned error can't find the container with id 0bdccf2d314989ccb03561168011a03ad2efa08de15198d5b91c5dc46e9fff20 Aug 04 19:45:51.624755 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:51.624721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" event={"ID":"34ec8e30-11bc-4059-a371-2023dcd0e08a","Type":"ContainerStarted","Data":"0bdccf2d314989ccb03561168011a03ad2efa08de15198d5b91c5dc46e9fff20"} Aug 04 19:45:52.631964 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:52.631922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" event={"ID":"34ec8e30-11bc-4059-a371-2023dcd0e08a","Type":"ContainerStarted","Data":"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b"} Aug 04 19:45:52.632386 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:52.632020 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:45:52.652804 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:52.652757 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" podStartSLOduration=2.005483324 podStartE2EDuration="2.652744451s" totalImagesPullingTime="647.261127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:52.651504457 +0000 UTC m=+1813.621530979" watchObservedRunningTime="2026-08-04 19:45:52.652744451 +0000 UTC m=+1813.622770971" Aug 04 19:45:53.689001 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:53.688967 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:45:58.643627 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:45:58.643595 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:46:13.797699 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:13.797665 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb"] Aug 04 19:46:13.801356 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:13.801335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:13.803856 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:13.803834 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:46:13.809652 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:13.808968 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb"] Aug 04 19:46:13.984478 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:13.984429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb\" (UID: \"b53046d8-f07d-424f-baba-f08558faa2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:14.085531 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.085443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb\" (UID: \"b53046d8-f07d-424f-baba-f08558faa2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:14.086192 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.086134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb\" (UID: \"b53046d8-f07d-424f-baba-f08558faa2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:14.113493 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.113461 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:14.251772 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.251745 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb"] Aug 04 19:46:14.253910 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:46:14.253886 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53046d8_f07d_424f_baba_f08558faa2f2.slice/crio-0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8 WatchSource:0}: Error finding container 0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8: Status 404 returned error can't find the container with id 0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8 Aug 04 19:46:14.752000 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.751911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" event={"ID":"b53046d8-f07d-424f-baba-f08558faa2f2","Type":"ContainerStarted","Data":"2c52e379e3784a7a0d24f59e539426a7b4caa560641957da8874e0b05d8dafa3"} Aug 04 19:46:14.752000 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.751970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" event={"ID":"b53046d8-f07d-424f-baba-f08558faa2f2","Type":"ContainerStarted","Data":"0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8"} Aug 04 19:46:14.769753 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:14.769704 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" podStartSLOduration=1.7696913589999999 podStartE2EDuration="1.769691359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:46:14.766633165 +0000 UTC m=+1835.736659684" watchObservedRunningTime="2026-08-04 19:46:14.769691359 +0000 UTC m=+1835.739717939" Aug 04 19:46:15.758489 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:15.758457 2567 generic.go:356] "Generic (PLEG): container finished" podID="b53046d8-f07d-424f-baba-f08558faa2f2" containerID="2c52e379e3784a7a0d24f59e539426a7b4caa560641957da8874e0b05d8dafa3" exitCode=0 Aug 04 19:46:15.758919 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:15.758550 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" event={"ID":"b53046d8-f07d-424f-baba-f08558faa2f2","Type":"ContainerDied","Data":"2c52e379e3784a7a0d24f59e539426a7b4caa560641957da8874e0b05d8dafa3"} Aug 04 19:46:16.896161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:16.896138 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:16.908621 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:16.908605 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca\") pod \"b53046d8-f07d-424f-baba-f08558faa2f2\" (UID: \"b53046d8-f07d-424f-baba-f08558faa2f2\") " Aug 04 19:46:16.908930 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:16.908909 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b53046d8-f07d-424f-baba-f08558faa2f2" (UID: "b53046d8-f07d-424f-baba-f08558faa2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:46:17.009413 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:17.009387 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53046d8-f07d-424f-baba-f08558faa2f2-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:46:17.770578 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:17.770495 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" Aug 04 19:46:17.770578 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:17.770528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb" event={"ID":"b53046d8-f07d-424f-baba-f08558faa2f2","Type":"ContainerDied","Data":"0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8"} Aug 04 19:46:17.770754 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:17.770582 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b38a850c4acaf503940761b11092959687f874749119a6f78f6bf8a9ae449a8" Aug 04 19:46:20.683948 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:20.683918 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:46:20.684416 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:20.684146 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" podUID="34ec8e30-11bc-4059-a371-2023dcd0e08a" containerName="maas-api" containerID="cri-o://b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b" gracePeriod=30 Aug 04 19:46:20.894362 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:20.894332 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb"] Aug 04 19:46:20.898006 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:20.897978 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3d6cfa2c-jwmkb"] Aug 04 19:46:20.943044 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:20.943021 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:46:21.038758 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.038732 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") pod \"34ec8e30-11bc-4059-a371-2023dcd0e08a\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " Aug 04 19:46:21.038910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.038775 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca\") pod \"34ec8e30-11bc-4059-a371-2023dcd0e08a\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " Aug 04 19:46:21.038910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.038816 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5hd5\" (UniqueName: \"kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5\") pod \"34ec8e30-11bc-4059-a371-2023dcd0e08a\" (UID: \"34ec8e30-11bc-4059-a371-2023dcd0e08a\") " Aug 04 19:46:21.039238 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.039207 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34ec8e30-11bc-4059-a371-2023dcd0e08a" (UID: "34ec8e30-11bc-4059-a371-2023dcd0e08a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:46:21.041294 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.041271 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34ec8e30-11bc-4059-a371-2023dcd0e08a" (UID: "34ec8e30-11bc-4059-a371-2023dcd0e08a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:46:21.041425 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.041400 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5" (OuterVolumeSpecName: "kube-api-access-q5hd5") pod "34ec8e30-11bc-4059-a371-2023dcd0e08a" (UID: "34ec8e30-11bc-4059-a371-2023dcd0e08a"). InnerVolumeSpecName "kube-api-access-q5hd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:46:21.139938 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.139903 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34ec8e30-11bc-4059-a371-2023dcd0e08a-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.139938 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.139931 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34ec8e30-11bc-4059-a371-2023dcd0e08a-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.139938 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.139944 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hd5\" (UniqueName: \"kubernetes.io/projected/34ec8e30-11bc-4059-a371-2023dcd0e08a-kube-api-access-q5hd5\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:46:21.504483 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.504449 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53046d8-f07d-424f-baba-f08558faa2f2" path="/var/lib/kubelet/pods/b53046d8-f07d-424f-baba-f08558faa2f2/volumes" Aug 04 19:46:21.794720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.794687 2567 generic.go:356] "Generic (PLEG): container finished" podID="34ec8e30-11bc-4059-a371-2023dcd0e08a" containerID="b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b" exitCode=0 Aug 04 19:46:21.795134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.794730 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" Aug 04 19:46:21.795134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.794777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" event={"ID":"34ec8e30-11bc-4059-a371-2023dcd0e08a","Type":"ContainerDied","Data":"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b"} Aug 04 19:46:21.795134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.794827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2" event={"ID":"34ec8e30-11bc-4059-a371-2023dcd0e08a","Type":"ContainerDied","Data":"0bdccf2d314989ccb03561168011a03ad2efa08de15198d5b91c5dc46e9fff20"} Aug 04 19:46:21.795134 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.794851 2567 scope.go:117] "RemoveContainer" containerID="b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b" Aug 04 19:46:21.804983 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.804964 2567 scope.go:117] "RemoveContainer" containerID="b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b" Aug 04 19:46:21.805265 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:46:21.805247 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b\": container with ID starting with b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b not found: ID does not exist" containerID="b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b" Aug 04 19:46:21.805329 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.805275 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b"} err="failed to get container status \"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b\": rpc error: code = NotFound desc = could not find container \"b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b\": container with ID starting with b34e41643a6d160518b49d99048ef749eb121021df06d8a3b6190e4a835d318b not found: ID does not exist" Aug 04 19:46:21.820031 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.820008 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:46:21.823546 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:21.823526 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3d6cfa2c-6bb4958f8c-vt2p2"] Aug 04 19:46:23.504206 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:23.504169 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ec8e30-11bc-4059-a371-2023dcd0e08a" path="/var/lib/kubelet/pods/34ec8e30-11bc-4059-a371-2023dcd0e08a/volumes" Aug 04 19:46:23.700529 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:23.700497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:46:36.596521 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:46:36.596485 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:48:06.693667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:06.693629 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:48:13.488158 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:13.488117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:48:23.750388 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750345 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750741 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53046d8-f07d-424f-baba-f08558faa2f2" containerName="revoke-keys" Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750754 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53046d8-f07d-424f-baba-f08558faa2f2" containerName="revoke-keys" Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750767 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ec8e30-11bc-4059-a371-2023dcd0e08a" containerName="maas-api" Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750773 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ec8e30-11bc-4059-a371-2023dcd0e08a" containerName="maas-api" Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750825 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ec8e30-11bc-4059-a371-2023dcd0e08a" containerName="maas-api" Aug 04 19:48:23.750867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.750837 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53046d8-f07d-424f-baba-f08558faa2f2" containerName="revoke-keys" Aug 04 19:48:23.753993 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.753971 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:23.756786 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.756762 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:48:23.756897 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.756851 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8129f458-serving-cert\"" Aug 04 19:48:23.757906 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.757885 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:48:23.757997 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.757916 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:48:23.765491 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.765471 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:48:23.911346 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.911309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:23.911523 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.911373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4p8\" (UniqueName: \"kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:23.911523 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:23.911464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.012837 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.012756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.012837 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.012813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.013096 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.012876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4p8\" (UniqueName: \"kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.013096 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:48:24.012921 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-serving-cert: secret "maas-api-e2e-mt-8129f458-serving-cert" not found Aug 04 19:48:24.013096 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:48:24.013006 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls podName:1688b128-667a-4435-8d23-8cbd4d3d9939 nodeName:}" failed. No retries permitted until 2026-08-04 19:48:24.512986599 +0000 UTC m=+1965.483013096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls") pod "maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" (UID: "1688b128-667a-4435-8d23-8cbd4d3d9939") : secret "maas-api-e2e-mt-8129f458-serving-cert" not found Aug 04 19:48:24.013433 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.013416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.021918 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.021893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4p8\" (UniqueName: \"kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.517261 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.517229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.519588 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.519561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") pod \"maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.665784 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.665746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:24.794241 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:24.794172 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:48:24.794560 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:48:24.794272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1688b128_667a_4435_8d23_8cbd4d3d9939.slice/crio-1773b797920d803a7c428d3f919d77f237f598ee3cb7bae8a86f611f2fc3e1a2 WatchSource:0}: Error finding container 1773b797920d803a7c428d3f919d77f237f598ee3cb7bae8a86f611f2fc3e1a2: Status 404 returned error can't find the container with id 1773b797920d803a7c428d3f919d77f237f598ee3cb7bae8a86f611f2fc3e1a2 Aug 04 19:48:25.480609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:25.480573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" event={"ID":"1688b128-667a-4435-8d23-8cbd4d3d9939","Type":"ContainerStarted","Data":"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542"} Aug 04 19:48:25.480781 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:25.480627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" event={"ID":"1688b128-667a-4435-8d23-8cbd4d3d9939","Type":"ContainerStarted","Data":"1773b797920d803a7c428d3f919d77f237f598ee3cb7bae8a86f611f2fc3e1a2"} Aug 04 19:48:25.480781 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:25.480717 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:25.497045 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:25.496964 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" podStartSLOduration=2.04762917 podStartE2EDuration="2.496950706s" totalImagesPullingTime="449.321536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:48:25.496195989 +0000 UTC m=+1966.466222509" watchObservedRunningTime="2026-08-04 19:48:25.496950706 +0000 UTC m=+1966.466977224" Aug 04 19:48:26.397983 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:26.397950 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:48:31.491485 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:31.491415 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:48:54.701683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.701645 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c"] Aug 04 19:48:54.705085 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.705070 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:54.707704 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.707682 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:48:54.713712 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.713691 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c"] Aug 04 19:48:54.741015 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.740991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8129f458-kch9c\" (UID: \"34b21318-4ac6-4c49-aebe-527935d24897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:54.841943 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.841922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8129f458-kch9c\" (UID: \"34b21318-4ac6-4c49-aebe-527935d24897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:54.842471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:54.842453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8129f458-kch9c\" (UID: \"34b21318-4ac6-4c49-aebe-527935d24897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:55.015778 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:55.015711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:55.346110 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:55.346088 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c"] Aug 04 19:48:55.645892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:55.645831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" event={"ID":"34b21318-4ac6-4c49-aebe-527935d24897","Type":"ContainerStarted","Data":"b09384132a909df648c8a5ce87c68ca6a4f8a9edbd3f0981d39b9826f1e52f38"} Aug 04 19:48:55.645892 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:55.645888 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" event={"ID":"34b21318-4ac6-4c49-aebe-527935d24897","Type":"ContainerStarted","Data":"f7cc5901ca6ca894f573d755f70d8ba0f41c1663a4871051831613061853ea11"} Aug 04 19:48:55.661386 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:55.661333 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" podStartSLOduration=1.6613149919999999 podStartE2EDuration="1.661314992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:48:55.658632969 +0000 UTC m=+1996.628659488" watchObservedRunningTime="2026-08-04 19:48:55.661314992 +0000 UTC m=+1996.631341512" Aug 04 19:48:56.654092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:56.654034 2567 generic.go:356] "Generic (PLEG): container finished" podID="34b21318-4ac6-4c49-aebe-527935d24897" containerID="b09384132a909df648c8a5ce87c68ca6a4f8a9edbd3f0981d39b9826f1e52f38" exitCode=0 Aug 04 19:48:56.654092 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:56.654083 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" event={"ID":"34b21318-4ac6-4c49-aebe-527935d24897","Type":"ContainerDied","Data":"b09384132a909df648c8a5ce87c68ca6a4f8a9edbd3f0981d39b9826f1e52f38"} Aug 04 19:48:57.791274 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:57.791251 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:57.863544 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:57.863519 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca\") pod \"34b21318-4ac6-4c49-aebe-527935d24897\" (UID: \"34b21318-4ac6-4c49-aebe-527935d24897\") " Aug 04 19:48:57.863843 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:57.863821 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34b21318-4ac6-4c49-aebe-527935d24897" (UID: "34b21318-4ac6-4c49-aebe-527935d24897"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:48:57.964183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:57.964124 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b21318-4ac6-4c49-aebe-527935d24897-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:48:58.666193 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:58.666164 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" Aug 04 19:48:58.666340 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:58.666208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c" event={"ID":"34b21318-4ac6-4c49-aebe-527935d24897","Type":"ContainerDied","Data":"f7cc5901ca6ca894f573d755f70d8ba0f41c1663a4871051831613061853ea11"} Aug 04 19:48:58.666340 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:48:58.666254 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cc5901ca6ca894f573d755f70d8ba0f41c1663a4871051831613061853ea11" Aug 04 19:49:05.211805 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.211771 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:49:05.212541 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.212510 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" podUID="1688b128-667a-4435-8d23-8cbd4d3d9939" containerName="maas-api" containerID="cri-o://972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542" gracePeriod=30 Aug 04 19:49:05.398095 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.398067 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c"] Aug 04 19:49:05.400253 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.400229 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8129f458-kch9c"] Aug 04 19:49:05.462722 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.462673 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:49:05.505446 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.505414 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b21318-4ac6-4c49-aebe-527935d24897" path="/var/lib/kubelet/pods/34b21318-4ac6-4c49-aebe-527935d24897/volumes" Aug 04 19:49:05.521343 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.521319 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") pod \"1688b128-667a-4435-8d23-8cbd4d3d9939\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " Aug 04 19:49:05.521481 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.521370 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca\") pod \"1688b128-667a-4435-8d23-8cbd4d3d9939\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " Aug 04 19:49:05.521481 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.521470 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn4p8\" (UniqueName: \"kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8\") pod \"1688b128-667a-4435-8d23-8cbd4d3d9939\" (UID: \"1688b128-667a-4435-8d23-8cbd4d3d9939\") " Aug 04 19:49:05.521969 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.521763 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1688b128-667a-4435-8d23-8cbd4d3d9939" (UID: "1688b128-667a-4435-8d23-8cbd4d3d9939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:49:05.523422 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.523392 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8" (OuterVolumeSpecName: "kube-api-access-hn4p8") pod "1688b128-667a-4435-8d23-8cbd4d3d9939" (UID: "1688b128-667a-4435-8d23-8cbd4d3d9939"). InnerVolumeSpecName "kube-api-access-hn4p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:05.523529 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.523436 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1688b128-667a-4435-8d23-8cbd4d3d9939" (UID: "1688b128-667a-4435-8d23-8cbd4d3d9939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:49:05.622830 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.622803 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1688b128-667a-4435-8d23-8cbd4d3d9939-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.622830 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.622825 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4p8\" (UniqueName: \"kubernetes.io/projected/1688b128-667a-4435-8d23-8cbd4d3d9939-kube-api-access-hn4p8\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.622984 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.622835 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1688b128-667a-4435-8d23-8cbd4d3d9939-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:49:05.704800 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.704775 2567 generic.go:356] "Generic (PLEG): container finished" podID="1688b128-667a-4435-8d23-8cbd4d3d9939" containerID="972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542" exitCode=0 Aug 04 19:49:05.704949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.704830 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" Aug 04 19:49:05.704949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.704867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" event={"ID":"1688b128-667a-4435-8d23-8cbd4d3d9939","Type":"ContainerDied","Data":"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542"} Aug 04 19:49:05.704949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.704923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r" event={"ID":"1688b128-667a-4435-8d23-8cbd4d3d9939","Type":"ContainerDied","Data":"1773b797920d803a7c428d3f919d77f237f598ee3cb7bae8a86f611f2fc3e1a2"} Aug 04 19:49:05.704949 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.704950 2567 scope.go:117] "RemoveContainer" containerID="972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542" Aug 04 19:49:05.714476 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.714457 2567 scope.go:117] "RemoveContainer" containerID="972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542" Aug 04 19:49:05.714742 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:49:05.714724 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542\": container with ID starting with 972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542 not found: ID does not exist" containerID="972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542" Aug 04 19:49:05.714785 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.714750 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542"} err="failed to get container status \"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542\": rpc error: code = NotFound desc = could not find container \"972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542\": container with ID starting with 972e852f962154c70bd7d7c7e27d0bc6344a7006347911f3bbe5add9df372542 not found: ID does not exist" Aug 04 19:49:05.735841 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.735821 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:49:05.739711 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:05.739692 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8129f458-78f4f8c78-2qj7r"] Aug 04 19:49:07.191710 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:07.191682 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:49:07.504694 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:07.504622 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1688b128-667a-4435-8d23-8cbd4d3d9939" path="/var/lib/kubelet/pods/1688b128-667a-4435-8d23-8cbd4d3d9939/volumes" Aug 04 19:49:46.787161 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:49:46.787133 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:50:39.668266 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:50:39.668236 2567 scope.go:117] "RemoveContainer" containerID="be703aae262074505ba0500fd041707c96c4bd02a7d986a13a00db1cf165d60c" Aug 04 19:51:29.589362 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:29.589330 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:51:32.343319 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343291 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:51:32.343661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343617 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b21318-4ac6-4c49-aebe-527935d24897" containerName="revoke-keys" Aug 04 19:51:32.343661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343627 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b21318-4ac6-4c49-aebe-527935d24897" containerName="revoke-keys" Aug 04 19:51:32.343661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343637 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1688b128-667a-4435-8d23-8cbd4d3d9939" containerName="maas-api" Aug 04 19:51:32.343661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343642 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1688b128-667a-4435-8d23-8cbd4d3d9939" containerName="maas-api" Aug 04 19:51:32.343827 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343690 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b21318-4ac6-4c49-aebe-527935d24897" containerName="revoke-keys" Aug 04 19:51:32.343827 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.343708 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1688b128-667a-4435-8d23-8cbd4d3d9939" containerName="maas-api" Aug 04 19:51:32.346807 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.346792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.353407 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.353383 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb61cc13-serving-cert\"" Aug 04 19:51:32.353489 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.353434 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:51:32.353534 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.353519 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:51:32.354368 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.354352 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:51:32.361867 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.361848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:51:32.460039 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.460013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.460178 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.460109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.460178 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.460155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmbw\" (UniqueName: \"kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.560720 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.560688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.560869 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.560730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmbw\" (UniqueName: \"kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.560869 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.560781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.561430 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.561409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.563372 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.563349 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.578475 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.578457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmbw\" (UniqueName: \"kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw\") pod \"maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.656884 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.656833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:32.796513 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.796485 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:51:32.797042 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:51:32.797019 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba21523_bae6_42df_817d_757c7df2b5e7.slice/crio-a84af715f6f2ab0a717047b2d985fd5faa932dfc6416e9e7c35dba804b27e190 WatchSource:0}: Error finding container a84af715f6f2ab0a717047b2d985fd5faa932dfc6416e9e7c35dba804b27e190: Status 404 returned error can't find the container with id a84af715f6f2ab0a717047b2d985fd5faa932dfc6416e9e7c35dba804b27e190 Aug 04 19:51:32.798719 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:32.798701 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:51:33.497600 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:33.497520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" event={"ID":"6ba21523-bae6-42df-817d-757c7df2b5e7","Type":"ContainerStarted","Data":"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89"} Aug 04 19:51:33.497600 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:33.497569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" event={"ID":"6ba21523-bae6-42df-817d-757c7df2b5e7","Type":"ContainerStarted","Data":"a84af715f6f2ab0a717047b2d985fd5faa932dfc6416e9e7c35dba804b27e190"} Aug 04 19:51:33.498106 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:33.497640 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:33.521928 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:33.521886 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" podStartSLOduration=1.106186336 podStartE2EDuration="1.521873113s" totalImagesPullingTime="415.686777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:51:33.519285606 +0000 UTC m=+2154.489312125" watchObservedRunningTime="2026-08-04 19:51:33.521873113 +0000 UTC m=+2154.491899677" Aug 04 19:51:39.508567 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:39.508541 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:51:39.688469 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:39.688442 2567 scope.go:117] "RemoveContainer" containerID="682e8f01179c82cdb5a11bf704c186270986dc789d7b97da0e540fb24d6f8d97" Aug 04 19:51:39.696524 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:39.696505 2567 scope.go:117] "RemoveContainer" containerID="836ac9c8534c1f58df12eb8cd3418223245f7973f2e120d4491766c5fa1a9b41" Aug 04 19:51:39.703964 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:39.703938 2567 scope.go:117] "RemoveContainer" containerID="bc0f92ba0a15ce37926a4d5b6dda4179729d25423d0049c459a39723814a4165" Aug 04 19:51:39.987883 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:51:39.987854 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:52:14.385826 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:14.385796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:52:32.094200 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:32.094163 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:52:39.728902 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:39.728871 2567 scope.go:117] "RemoveContainer" containerID="2c52e379e3784a7a0d24f59e539426a7b4caa560641957da8874e0b05d8dafa3" Aug 04 19:52:40.968681 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:40.968651 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r"] Aug 04 19:52:40.973623 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:40.973601 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:40.979127 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:40.979105 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:52:40.982216 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:40.982195 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r"] Aug 04 19:52:41.105773 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.105750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r\" (UID: \"1e70fa55-3b79-4772-b278-7e8991330b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:41.206254 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.206231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r\" (UID: \"1e70fa55-3b79-4772-b278-7e8991330b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:41.206733 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.206716 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r\" (UID: \"1e70fa55-3b79-4772-b278-7e8991330b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:41.285512 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.285489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:41.407643 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.407613 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r"] Aug 04 19:52:41.409142 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:52:41.409116 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e70fa55_3b79_4772_b278_7e8991330b45.slice/crio-6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1 WatchSource:0}: Error finding container 6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1: Status 404 returned error can't find the container with id 6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1 Aug 04 19:52:41.847036 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.847007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" event={"ID":"1e70fa55-3b79-4772-b278-7e8991330b45","Type":"ContainerStarted","Data":"4b10e23c20bc628ceb9f4cfa6472aa9b7f5f186db76ad4183afdd8355a70843b"} Aug 04 19:52:41.847165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.847107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" event={"ID":"1e70fa55-3b79-4772-b278-7e8991330b45","Type":"ContainerStarted","Data":"6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1"} Aug 04 19:52:41.864793 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:41.864748 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" podStartSLOduration=1.8647361820000001 podStartE2EDuration="1.864736182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:52:41.862207062 +0000 UTC m=+2222.832233581" watchObservedRunningTime="2026-08-04 19:52:41.864736182 +0000 UTC m=+2222.834762701" Aug 04 19:52:42.853784 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:42.853741 2567 generic.go:356] "Generic (PLEG): container finished" podID="1e70fa55-3b79-4772-b278-7e8991330b45" containerID="4b10e23c20bc628ceb9f4cfa6472aa9b7f5f186db76ad4183afdd8355a70843b" exitCode=0 Aug 04 19:52:42.854236 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:42.853812 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" event={"ID":"1e70fa55-3b79-4772-b278-7e8991330b45","Type":"ContainerDied","Data":"4b10e23c20bc628ceb9f4cfa6472aa9b7f5f186db76ad4183afdd8355a70843b"} Aug 04 19:52:43.986989 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:43.986966 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:44.131393 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.131329 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca\") pod \"1e70fa55-3b79-4772-b278-7e8991330b45\" (UID: \"1e70fa55-3b79-4772-b278-7e8991330b45\") " Aug 04 19:52:44.131637 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.131616 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e70fa55-3b79-4772-b278-7e8991330b45" (UID: "1e70fa55-3b79-4772-b278-7e8991330b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:52:44.231845 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.231818 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e70fa55-3b79-4772-b278-7e8991330b45-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:52:44.865377 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.865350 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" Aug 04 19:52:44.865377 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.865365 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r" event={"ID":"1e70fa55-3b79-4772-b278-7e8991330b45","Type":"ContainerDied","Data":"6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1"} Aug 04 19:52:44.865596 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:44.865410 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ecfb0ab88e00454c2b8631e16b105237b61a2f83826638ebf8d88cb1f6d71f1" Aug 04 19:52:54.086674 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:54.086640 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:52:56.416987 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.416954 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:52:56.417457 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.417379 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" podUID="6ba21523-bae6-42df-817d-757c7df2b5e7" containerName="maas-api" containerID="cri-o://823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89" gracePeriod=30 Aug 04 19:52:56.600721 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.600649 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r"] Aug 04 19:52:56.603467 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.603440 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb61cc13-njz6r"] Aug 04 19:52:56.663022 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.663001 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:52:56.828974 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.828941 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmbw\" (UniqueName: \"kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw\") pod \"6ba21523-bae6-42df-817d-757c7df2b5e7\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " Aug 04 19:52:56.829168 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.828996 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls\") pod \"6ba21523-bae6-42df-817d-757c7df2b5e7\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " Aug 04 19:52:56.829168 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.829087 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca\") pod \"6ba21523-bae6-42df-817d-757c7df2b5e7\" (UID: \"6ba21523-bae6-42df-817d-757c7df2b5e7\") " Aug 04 19:52:56.829459 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.829433 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ba21523-bae6-42df-817d-757c7df2b5e7" (UID: "6ba21523-bae6-42df-817d-757c7df2b5e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:52:56.831197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.831159 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ba21523-bae6-42df-817d-757c7df2b5e7" (UID: "6ba21523-bae6-42df-817d-757c7df2b5e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:52:56.831471 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.831451 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw" (OuterVolumeSpecName: "kube-api-access-dtmbw") pod "6ba21523-bae6-42df-817d-757c7df2b5e7" (UID: "6ba21523-bae6-42df-817d-757c7df2b5e7"). InnerVolumeSpecName "kube-api-access-dtmbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:52:56.929992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.929967 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmbw\" (UniqueName: \"kubernetes.io/projected/6ba21523-bae6-42df-817d-757c7df2b5e7-kube-api-access-dtmbw\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.929992 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.929991 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ba21523-bae6-42df-817d-757c7df2b5e7-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.930144 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.930005 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba21523-bae6-42df-817d-757c7df2b5e7-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:52:56.934750 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.934725 2567 generic.go:356] "Generic (PLEG): container finished" podID="6ba21523-bae6-42df-817d-757c7df2b5e7" containerID="823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89" exitCode=0 Aug 04 19:52:56.934871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.934768 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" Aug 04 19:52:56.934871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.934813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" event={"ID":"6ba21523-bae6-42df-817d-757c7df2b5e7","Type":"ContainerDied","Data":"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89"} Aug 04 19:52:56.934871 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.934860 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62" event={"ID":"6ba21523-bae6-42df-817d-757c7df2b5e7","Type":"ContainerDied","Data":"a84af715f6f2ab0a717047b2d985fd5faa932dfc6416e9e7c35dba804b27e190"} Aug 04 19:52:56.934994 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.934885 2567 scope.go:117] "RemoveContainer" containerID="823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89" Aug 04 19:52:56.944217 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.944202 2567 scope.go:117] "RemoveContainer" containerID="823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89" Aug 04 19:52:56.944462 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:52:56.944440 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89\": container with ID starting with 823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89 not found: ID does not exist" containerID="823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89" Aug 04 19:52:56.944513 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.944475 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89"} err="failed to get container status \"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89\": rpc error: code = NotFound desc = could not find container \"823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89\": container with ID starting with 823e6dcf08276be24a531ca543e7957c912002d6b0684d78824a0df275c04e89 not found: ID does not exist" Aug 04 19:52:56.955959 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.955940 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:52:56.959910 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:56.959892 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb61cc13-77dfff7b76-65r62"] Aug 04 19:52:57.504207 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:57.504178 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e70fa55-3b79-4772-b278-7e8991330b45" path="/var/lib/kubelet/pods/1e70fa55-3b79-4772-b278-7e8991330b45/volumes" Aug 04 19:52:57.504572 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:52:57.504556 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba21523-bae6-42df-817d-757c7df2b5e7" path="/var/lib/kubelet/pods/6ba21523-bae6-42df-817d-757c7df2b5e7/volumes" Aug 04 19:53:02.490428 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:53:02.490341 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:53:49.796419 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:53:49.796380 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:54:01.894562 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:01.894522 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:54:08.893235 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:08.893202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:54:11.088421 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:11.088393 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:54:57.326315 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326282 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:54:57.326936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326791 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e70fa55-3b79-4772-b278-7e8991330b45" containerName="revoke-keys" Aug 04 19:54:57.326936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326806 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e70fa55-3b79-4772-b278-7e8991330b45" containerName="revoke-keys" Aug 04 19:54:57.326936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326834 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba21523-bae6-42df-817d-757c7df2b5e7" containerName="maas-api" Aug 04 19:54:57.326936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326842 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba21523-bae6-42df-817d-757c7df2b5e7" containerName="maas-api" Aug 04 19:54:57.326936 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326929 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba21523-bae6-42df-817d-757c7df2b5e7" containerName="maas-api" Aug 04 19:54:57.327165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.326942 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e70fa55-3b79-4772-b278-7e8991330b45" containerName="revoke-keys" Aug 04 19:54:57.330090 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.330068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.332551 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.332526 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-48bed7-serving-cert\"" Aug 04 19:54:57.332683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.332625 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:54:57.332683 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.332627 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:54:57.333500 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.333482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:54:57.339990 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.339964 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:54:57.389952 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.389921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.390129 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.389969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jwf\" (UniqueName: \"kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.390129 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.389995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.491260 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.491229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.491406 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.491274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jwf\" (UniqueName: \"kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.491406 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.491295 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.491406 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:54:57.491367 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-serving-cert: secret "maas-api-e2e-inf-a-48bed7-serving-cert" not found Aug 04 19:54:57.491540 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:54:57.491437 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls podName:533b12e1-1f6e-4cf5-890d-0a5a7c39a29e nodeName:}" failed. No retries permitted until 2026-08-04 19:54:57.991421875 +0000 UTC m=+2358.961448371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls") pod "maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" (UID: "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e") : secret "maas-api-e2e-inf-a-48bed7-serving-cert" not found Aug 04 19:54:57.491834 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.491818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.500269 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.500245 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jwf\" (UniqueName: \"kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.995438 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.995402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:57.997764 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:57.997740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:58.243700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:58.243661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:58.371745 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:58.371718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:54:58.373075 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:54:58.373032 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b12e1_1f6e_4cf5_890d_0a5a7c39a29e.slice/crio-8d673eb54f4f86518070a2cf5c48ecf989dcff9e8115c3a3626189695a81fee2 WatchSource:0}: Error finding container 8d673eb54f4f86518070a2cf5c48ecf989dcff9e8115c3a3626189695a81fee2: Status 404 returned error can't find the container with id 8d673eb54f4f86518070a2cf5c48ecf989dcff9e8115c3a3626189695a81fee2 Aug 04 19:54:58.569165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:58.569129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" event={"ID":"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e","Type":"ContainerStarted","Data":"8d673eb54f4f86518070a2cf5c48ecf989dcff9e8115c3a3626189695a81fee2"} Aug 04 19:54:59.575444 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:59.575415 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" event={"ID":"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e","Type":"ContainerStarted","Data":"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871"} Aug 04 19:54:59.575829 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:59.575519 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:54:59.591019 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:54:59.590971 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" podStartSLOduration=2.079805622 podStartE2EDuration="2.590956605s" totalImagesPullingTime="511.150983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:54:59.590422928 +0000 UTC m=+2360.560449465" watchObservedRunningTime="2026-08-04 19:54:59.590956605 +0000 UTC m=+2360.560983114" Aug 04 19:55:00.188822 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:00.188786 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:55:05.586399 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:05.586372 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:55:25.609825 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.609788 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:55:25.613307 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.613285 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.617137 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.617114 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ddb738-serving-cert\"" Aug 04 19:55:25.637631 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.637607 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:55:25.714374 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.714348 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.714536 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.714379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfz6v\" (UniqueName: \"kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.714536 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.714477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.814832 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.814801 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.814971 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.814841 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfz6v\" (UniqueName: \"kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.814971 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.814903 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.815123 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:55:25.815078 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-serving-cert: secret "maas-api-e2e-inf-b-ddb738-serving-cert" not found Aug 04 19:55:25.815190 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:55:25.815155 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls podName:9de87787-e4fd-4791-83f4-4fb79d4dfd2a nodeName:}" failed. No retries permitted until 2026-08-04 19:55:26.315134711 +0000 UTC m=+2387.285161213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls") pod "maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" (UID: "9de87787-e4fd-4791-83f4-4fb79d4dfd2a") : secret "maas-api-e2e-inf-b-ddb738-serving-cert" not found Aug 04 19:55:25.815400 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.815379 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:25.825704 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:25.825686 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfz6v\" (UniqueName: \"kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:26.319643 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:26.319612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:26.321876 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:26.321858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:26.524103 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:26.524074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:26.652163 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:26.652141 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:55:26.654186 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:55:26.654162 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de87787_e4fd_4791_83f4_4fb79d4dfd2a.slice/crio-ed4be106c72ec703cc37409a767bd22ba2b7f3a41e5468f17af477792aa465ef WatchSource:0}: Error finding container ed4be106c72ec703cc37409a767bd22ba2b7f3a41e5468f17af477792aa465ef: Status 404 returned error can't find the container with id ed4be106c72ec703cc37409a767bd22ba2b7f3a41e5468f17af477792aa465ef Aug 04 19:55:26.721920 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:26.721892 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" event={"ID":"9de87787-e4fd-4791-83f4-4fb79d4dfd2a","Type":"ContainerStarted","Data":"ed4be106c72ec703cc37409a767bd22ba2b7f3a41e5468f17af477792aa465ef"} Aug 04 19:55:27.728637 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:27.728601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" event={"ID":"9de87787-e4fd-4791-83f4-4fb79d4dfd2a","Type":"ContainerStarted","Data":"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c"} Aug 04 19:55:27.729003 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:27.728668 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:27.747272 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:27.747222 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" podStartSLOduration=2.373182944 podStartE2EDuration="2.7472059s" totalImagesPullingTime="374.022956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:55:27.746136849 +0000 UTC m=+2388.716163369" watchObservedRunningTime="2026-08-04 19:55:27.7472059 +0000 UTC m=+2388.717232419" Aug 04 19:55:28.490029 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:28.489994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:55:33.739356 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:33.739326 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:55:39.770981 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:55:39.770947 2567 scope.go:117] "RemoveContainer" containerID="b09384132a909df648c8a5ce87c68ca6a4f8a9edbd3f0981d39b9826f1e52f38" Aug 04 19:56:10.087506 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:56:10.087474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:56:18.593463 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:56:18.593426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:56:29.593268 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:56:29.593236 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:56:41.021133 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:56:41.021099 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:57:10.202243 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.202166 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5"] Aug 04 19:57:10.207821 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.207785 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:10.211694 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.211668 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:57:10.214101 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.214077 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5"] Aug 04 19:57:10.359162 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.359134 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5\" (UID: \"a12452bf-3281-49e9-a15b-f1168dd44b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:10.460412 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.460342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5\" (UID: \"a12452bf-3281-49e9-a15b-f1168dd44b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:10.460945 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.460926 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5\" (UID: \"a12452bf-3281-49e9-a15b-f1168dd44b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:10.520837 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.520812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:10.655783 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:10.655757 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5"] Aug 04 19:57:10.658191 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:57:10.658155 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12452bf_3281_49e9_a15b_f1168dd44b57.slice/crio-bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e WatchSource:0}: Error finding container bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e: Status 404 returned error can't find the container with id bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e Aug 04 19:57:11.276023 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:11.275944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" event={"ID":"a12452bf-3281-49e9-a15b-f1168dd44b57","Type":"ContainerStarted","Data":"0543af2ff66b7cb580bd3e6e5f740971aa697dd86f1b2545d579d1f3673216eb"} Aug 04 19:57:11.276402 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:11.276030 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" event={"ID":"a12452bf-3281-49e9-a15b-f1168dd44b57","Type":"ContainerStarted","Data":"bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e"} Aug 04 19:57:11.293197 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:11.293155 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" podStartSLOduration=1.2931419069999999 podStartE2EDuration="1.293141907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:57:11.291832848 +0000 UTC m=+2492.261859368" watchObservedRunningTime="2026-08-04 19:57:11.293141907 +0000 UTC m=+2492.263168426" Aug 04 19:57:12.282603 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:12.282569 2567 generic.go:356] "Generic (PLEG): container finished" podID="a12452bf-3281-49e9-a15b-f1168dd44b57" containerID="0543af2ff66b7cb580bd3e6e5f740971aa697dd86f1b2545d579d1f3673216eb" exitCode=0 Aug 04 19:57:12.282980 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:12.282653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" event={"ID":"a12452bf-3281-49e9-a15b-f1168dd44b57","Type":"ContainerDied","Data":"0543af2ff66b7cb580bd3e6e5f740971aa697dd86f1b2545d579d1f3673216eb"} Aug 04 19:57:13.424620 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:13.424592 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:13.588354 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:13.588287 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca\") pod \"a12452bf-3281-49e9-a15b-f1168dd44b57\" (UID: \"a12452bf-3281-49e9-a15b-f1168dd44b57\") " Aug 04 19:57:13.588604 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:13.588582 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a12452bf-3281-49e9-a15b-f1168dd44b57" (UID: "a12452bf-3281-49e9-a15b-f1168dd44b57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:13.689690 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:13.689661 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12452bf-3281-49e9-a15b-f1168dd44b57-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:13.998951 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:13.998872 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:57:14.293857 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:14.293833 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" Aug 04 19:57:14.294013 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:14.293824 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5" event={"ID":"a12452bf-3281-49e9-a15b-f1168dd44b57","Type":"ContainerDied","Data":"bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e"} Aug 04 19:57:14.294013 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:14.293953 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd872851fa63d7936af795f9a0a509401cf9aef097c132a75538aac282044b1e" Aug 04 19:57:20.596976 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.596943 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:57:20.599416 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.597213 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" podUID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" containerName="maas-api" containerID="cri-o://6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871" gracePeriod=30 Aug 04 19:57:20.805746 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.805719 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5"] Aug 04 19:57:20.808965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.808935 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48bed7-zc7c5"] Aug 04 19:57:20.851605 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.851553 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:57:20.949259 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.949237 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca\") pod \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " Aug 04 19:57:20.949402 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.949306 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6jwf\" (UniqueName: \"kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf\") pod \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " Aug 04 19:57:20.949402 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.949327 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") pod \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\" (UID: \"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e\") " Aug 04 19:57:20.949600 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.949577 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" (UID: "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:20.951433 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.951410 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" (UID: "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:57:20.951530 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:20.951427 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf" (OuterVolumeSpecName: "kube-api-access-l6jwf") pod "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" (UID: "533b12e1-1f6e-4cf5-890d-0a5a7c39a29e"). InnerVolumeSpecName "kube-api-access-l6jwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:21.049970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.049943 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.049970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.049966 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jwf\" (UniqueName: \"kubernetes.io/projected/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-kube-api-access-l6jwf\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.049970 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.049977 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:21.333659 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.333627 2567 generic.go:356] "Generic (PLEG): container finished" podID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" containerID="6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871" exitCode=0 Aug 04 19:57:21.333831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.333676 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" Aug 04 19:57:21.333831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.333709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" event={"ID":"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e","Type":"ContainerDied","Data":"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871"} Aug 04 19:57:21.333831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.333770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t" event={"ID":"533b12e1-1f6e-4cf5-890d-0a5a7c39a29e","Type":"ContainerDied","Data":"8d673eb54f4f86518070a2cf5c48ecf989dcff9e8115c3a3626189695a81fee2"} Aug 04 19:57:21.333831 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.333800 2567 scope.go:117] "RemoveContainer" containerID="6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871" Aug 04 19:57:21.344500 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.344484 2567 scope.go:117] "RemoveContainer" containerID="6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871" Aug 04 19:57:21.344799 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:57:21.344778 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871\": container with ID starting with 6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871 not found: ID does not exist" containerID="6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871" Aug 04 19:57:21.344857 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.344807 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871"} err="failed to get container status \"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871\": rpc error: code = NotFound desc = could not find container \"6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871\": container with ID starting with 6dd06714db83a09bc669ede5ad478de0e796cd8f4cc183cbd83201af19342871 not found: ID does not exist" Aug 04 19:57:21.356589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.356568 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:57:21.360256 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.360236 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48bed7-666b79cc57-gmz5t"] Aug 04 19:57:21.504298 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.504268 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" path="/var/lib/kubelet/pods/533b12e1-1f6e-4cf5-890d-0a5a7c39a29e/volumes" Aug 04 19:57:21.504589 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:21.504576 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12452bf-3281-49e9-a15b-f1168dd44b57" path="/var/lib/kubelet/pods/a12452bf-3281-49e9-a15b-f1168dd44b57/volumes" Aug 04 19:57:22.993676 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:22.993642 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:57:27.710111 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710080 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds"] Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710438 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12452bf-3281-49e9-a15b-f1168dd44b57" containerName="revoke-keys" Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710450 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12452bf-3281-49e9-a15b-f1168dd44b57" containerName="revoke-keys" Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710461 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" containerName="maas-api" Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710467 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" containerName="maas-api" Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710524 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="533b12e1-1f6e-4cf5-890d-0a5a7c39a29e" containerName="maas-api" Aug 04 19:57:27.710560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.710533 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12452bf-3281-49e9-a15b-f1168dd44b57" containerName="revoke-keys" Aug 04 19:57:27.714819 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.714803 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:27.717558 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.717515 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:57:27.720848 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.720826 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds"] Aug 04 19:57:27.799479 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.799443 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds\" (UID: \"24c8858d-2aa0-4389-8769-916e9296bae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:27.900609 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.900581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds\" (UID: \"24c8858d-2aa0-4389-8769-916e9296bae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:27.901183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:27.901165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds\" (UID: \"24c8858d-2aa0-4389-8769-916e9296bae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:28.026870 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:28.026810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:28.372414 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:28.370748 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds"] Aug 04 19:57:28.379628 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:28.379601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" event={"ID":"24c8858d-2aa0-4389-8769-916e9296bae2","Type":"ContainerStarted","Data":"9a608b5eeeaff145e7e47c7d3b3881e1b303feb64a23d4a5bbe3e02b82e8711d"} Aug 04 19:57:29.386739 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:29.386706 2567 generic.go:356] "Generic (PLEG): container finished" podID="24c8858d-2aa0-4389-8769-916e9296bae2" containerID="02a79452088bae389ad5a3b92edcae4ef899525181a9fd381fd1b60a46d74952" exitCode=0 Aug 04 19:57:29.387213 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:29.386754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" event={"ID":"24c8858d-2aa0-4389-8769-916e9296bae2","Type":"ContainerDied","Data":"02a79452088bae389ad5a3b92edcae4ef899525181a9fd381fd1b60a46d74952"} Aug 04 19:57:30.519530 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:30.519502 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:30.622874 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:30.622847 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca\") pod \"24c8858d-2aa0-4389-8769-916e9296bae2\" (UID: \"24c8858d-2aa0-4389-8769-916e9296bae2\") " Aug 04 19:57:30.623165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:30.623146 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24c8858d-2aa0-4389-8769-916e9296bae2" (UID: "24c8858d-2aa0-4389-8769-916e9296bae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:30.723503 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:30.723449 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c8858d-2aa0-4389-8769-916e9296bae2-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:31.398994 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:31.398929 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" Aug 04 19:57:31.399160 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:31.398926 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds" event={"ID":"24c8858d-2aa0-4389-8769-916e9296bae2","Type":"ContainerDied","Data":"9a608b5eeeaff145e7e47c7d3b3881e1b303feb64a23d4a5bbe3e02b82e8711d"} Aug 04 19:57:31.399160 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:31.399077 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a608b5eeeaff145e7e47c7d3b3881e1b303feb64a23d4a5bbe3e02b82e8711d" Aug 04 19:57:38.114494 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.114458 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:57:38.115183 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.114684 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" podUID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" containerName="maas-api" containerID="cri-o://2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c" gracePeriod=30 Aug 04 19:57:38.304775 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.304747 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds"] Aug 04 19:57:38.306943 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.306923 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ddb738-cjrds"] Aug 04 19:57:38.366108 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.366022 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:57:38.439042 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.439011 2567 generic.go:356] "Generic (PLEG): container finished" podID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" containerID="2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c" exitCode=0 Aug 04 19:57:38.439042 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.439065 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" event={"ID":"9de87787-e4fd-4791-83f4-4fb79d4dfd2a","Type":"ContainerDied","Data":"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c"} Aug 04 19:57:38.439239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.439080 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" Aug 04 19:57:38.439239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.439121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb" event={"ID":"9de87787-e4fd-4791-83f4-4fb79d4dfd2a","Type":"ContainerDied","Data":"ed4be106c72ec703cc37409a767bd22ba2b7f3a41e5468f17af477792aa465ef"} Aug 04 19:57:38.439239 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.439151 2567 scope.go:117] "RemoveContainer" containerID="2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c" Aug 04 19:57:38.447638 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.447618 2567 scope.go:117] "RemoveContainer" containerID="2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c" Aug 04 19:57:38.447901 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:57:38.447883 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c\": container with ID starting with 2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c not found: ID does not exist" containerID="2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c" Aug 04 19:57:38.447952 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.447910 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c"} err="failed to get container status \"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c\": rpc error: code = NotFound desc = could not find container \"2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c\": container with ID starting with 2b6a6181ac9bef3f74cf13408ed14439a664474f992db448fc654b1d10bdcb7c not found: ID does not exist" Aug 04 19:57:38.487334 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.487312 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfz6v\" (UniqueName: \"kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v\") pod \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " Aug 04 19:57:38.487472 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.487359 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca\") pod \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " Aug 04 19:57:38.487472 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.487446 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") pod \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\" (UID: \"9de87787-e4fd-4791-83f4-4fb79d4dfd2a\") " Aug 04 19:57:38.487786 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.487759 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9de87787-e4fd-4791-83f4-4fb79d4dfd2a" (UID: "9de87787-e4fd-4791-83f4-4fb79d4dfd2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:38.489406 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.489382 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v" (OuterVolumeSpecName: "kube-api-access-dfz6v") pod "9de87787-e4fd-4791-83f4-4fb79d4dfd2a" (UID: "9de87787-e4fd-4791-83f4-4fb79d4dfd2a"). InnerVolumeSpecName "kube-api-access-dfz6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:38.489573 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.489549 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9de87787-e4fd-4791-83f4-4fb79d4dfd2a" (UID: "9de87787-e4fd-4791-83f4-4fb79d4dfd2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:57:38.588737 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.588713 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.588737 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.588737 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfz6v\" (UniqueName: \"kubernetes.io/projected/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-kube-api-access-dfz6v\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.588874 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.588747 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de87787-e4fd-4791-83f4-4fb79d4dfd2a-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:57:38.760607 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.760582 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:57:38.765463 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:38.765440 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddb738-556f6f665d-hs7lb"] Aug 04 19:57:39.503967 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:39.503939 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c8858d-2aa0-4389-8769-916e9296bae2" path="/var/lib/kubelet/pods/24c8858d-2aa0-4389-8769-916e9296bae2/volumes" Aug 04 19:57:39.504336 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:39.504317 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" path="/var/lib/kubelet/pods/9de87787-e4fd-4791-83f4-4fb79d4dfd2a/volumes" Aug 04 19:57:40.394667 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:40.394628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:57:49.158096 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158059 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158389 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" containerName="maas-api" Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158400 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" containerName="maas-api" Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158414 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c8858d-2aa0-4389-8769-916e9296bae2" containerName="revoke-keys" Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158419 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c8858d-2aa0-4389-8769-916e9296bae2" containerName="revoke-keys" Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158479 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de87787-e4fd-4791-83f4-4fb79d4dfd2a" containerName="maas-api" Aug 04 19:57:49.158564 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.158487 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c8858d-2aa0-4389-8769-916e9296bae2" containerName="revoke-keys" Aug 04 19:57:49.161426 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.161408 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.163728 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.163702 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4feefd-serving-cert\"" Aug 04 19:57:49.163860 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.163703 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:57:49.164765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.164749 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:57:49.164765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.164757 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:57:49.170227 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.170207 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:57:49.278847 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.278819 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.278991 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.278854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6m26\" (UniqueName: \"kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.278991 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.278881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.379494 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.379465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.379616 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.379503 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6m26\" (UniqueName: \"kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.379616 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.379527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.379738 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:57:49.379630 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-serving-cert: secret "maas-api-e2e-api-a-4feefd-serving-cert" not found Aug 04 19:57:49.379738 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:57:49.379719 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls podName:30e405a5-d80d-4de1-bfdc-88891e6be9de nodeName:}" failed. No retries permitted until 2026-08-04 19:57:49.879693752 +0000 UTC m=+2530.849720257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls") pod "maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" (UID: "30e405a5-d80d-4de1-bfdc-88891e6be9de") : secret "maas-api-e2e-api-a-4feefd-serving-cert" not found Aug 04 19:57:49.380255 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.380237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.387801 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.387773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6m26\" (UniqueName: \"kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.884071 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.884018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:49.886328 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:49.886308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") pod \"maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:50.072489 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:50.072462 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:50.208822 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:50.208796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:57:50.210291 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:57:50.210263 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e405a5_d80d_4de1_bfdc_88891e6be9de.slice/crio-ea41442bb4ed38a6fd1cdffa1116960749fde5655d79b5d24e5ec5b062c00208 WatchSource:0}: Error finding container ea41442bb4ed38a6fd1cdffa1116960749fde5655d79b5d24e5ec5b062c00208: Status 404 returned error can't find the container with id ea41442bb4ed38a6fd1cdffa1116960749fde5655d79b5d24e5ec5b062c00208 Aug 04 19:57:50.211591 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:50.211574 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:57:50.504406 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:50.504321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" event={"ID":"30e405a5-d80d-4de1-bfdc-88891e6be9de","Type":"ContainerStarted","Data":"ea41442bb4ed38a6fd1cdffa1116960749fde5655d79b5d24e5ec5b062c00208"} Aug 04 19:57:51.510002 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:51.509968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" event={"ID":"30e405a5-d80d-4de1-bfdc-88891e6be9de","Type":"ContainerStarted","Data":"bccf77065b07608b7efdfdff3f56cfcffb1a82f32dae36ae9c685b4164d785c5"} Aug 04 19:57:51.510469 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:51.510075 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:57:51.527202 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:51.527156 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" podStartSLOduration=2.010507809 podStartE2EDuration="2.527143399s" totalImagesPullingTime="516.63559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:57:51.52538143 +0000 UTC m=+2532.495407948" watchObservedRunningTime="2026-08-04 19:57:51.527143399 +0000 UTC m=+2532.497169916" Aug 04 19:57:51.993879 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:51.993830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:57:57.521291 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:57:57.521265 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:58:18.351392 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.351355 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:58:18.355246 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.355225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.357579 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.357554 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2e00c3-serving-cert\"" Aug 04 19:58:18.365273 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.365249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:58:18.512479 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.512448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.512634 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.512490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzr6\" (UniqueName: \"kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.512634 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.512620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.613767 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.613706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.613767 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.613737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.613767 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.613763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzr6\" (UniqueName: \"kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.614362 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.614340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.616103 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.616079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.622180 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.622156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzr6\" (UniqueName: \"kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6\") pod \"maas-api-e2e-api-b-2e00c3-656b87845b-srr67\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.666914 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.666893 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:18.802894 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:18.802859 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:58:18.804869 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:58:18.804835 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1572c659_09bb_4830_9f64_b9d948b6d55b.slice/crio-2aff7d73818329c6d0483f2d6eb1de951ace5710e300f3d8e5f90893df9de8dc WatchSource:0}: Error finding container 2aff7d73818329c6d0483f2d6eb1de951ace5710e300f3d8e5f90893df9de8dc: Status 404 returned error can't find the container with id 2aff7d73818329c6d0483f2d6eb1de951ace5710e300f3d8e5f90893df9de8dc Aug 04 19:58:19.659983 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:19.659948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" event={"ID":"1572c659-09bb-4830-9f64-b9d948b6d55b","Type":"ContainerStarted","Data":"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f"} Aug 04 19:58:19.660378 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:19.659999 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" event={"ID":"1572c659-09bb-4830-9f64-b9d948b6d55b","Type":"ContainerStarted","Data":"2aff7d73818329c6d0483f2d6eb1de951ace5710e300f3d8e5f90893df9de8dc"} Aug 04 19:58:19.660378 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:19.660039 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:19.680339 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:19.680292 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" podStartSLOduration=1.211711172 podStartE2EDuration="1.680277928s" totalImagesPullingTime="468.566756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:58:19.678032303 +0000 UTC m=+2560.648058822" watchObservedRunningTime="2026-08-04 19:58:19.680277928 +0000 UTC m=+2560.650304447" Aug 04 19:58:21.589466 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:21.589434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:58:25.670333 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:25.670307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:58:45.166693 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.166662 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq"] Aug 04 19:58:45.171114 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.171098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:45.173656 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.173635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:58:45.178106 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.178087 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq"] Aug 04 19:58:45.215573 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.215549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4feefd-896vq\" (UID: \"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:45.316028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.315997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4feefd-896vq\" (UID: \"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:45.316574 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.316557 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4feefd-896vq\" (UID: \"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:45.483152 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.483088 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:45.616395 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.616373 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq"] Aug 04 19:58:45.618152 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:58:45.618123 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7167f9c2_8a1b_4e11_92fe_5a6f22b3885e.slice/crio-45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe WatchSource:0}: Error finding container 45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe: Status 404 returned error can't find the container with id 45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe Aug 04 19:58:45.796681 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:45.796651 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" event={"ID":"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e","Type":"ContainerStarted","Data":"45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe"} Aug 04 19:58:46.803460 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:46.803424 2567 generic.go:356] "Generic (PLEG): container finished" podID="7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" containerID="f70473a9a77eebce6d8aeac6d1eece0e8f26a7be614dd3fe3dfc7e44e531d7ec" exitCode=0 Aug 04 19:58:46.804074 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:46.803506 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" event={"ID":"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e","Type":"ContainerDied","Data":"f70473a9a77eebce6d8aeac6d1eece0e8f26a7be614dd3fe3dfc7e44e531d7ec"} Aug 04 19:58:47.941010 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:47.940987 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:48.043663 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.043639 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca\") pod \"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e\" (UID: \"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e\") " Aug 04 19:58:48.043972 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.043950 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" (UID: "7167f9c2-8a1b-4e11-92fe-5a6f22b3885e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:58:48.144877 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.144805 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:58:48.815630 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.815603 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" Aug 04 19:58:48.815805 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.815608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq" event={"ID":"7167f9c2-8a1b-4e11-92fe-5a6f22b3885e","Type":"ContainerDied","Data":"45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe"} Aug 04 19:58:48.815805 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:48.815726 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a57808eee920f2775c0ee12ec2c9a5cae5f225e0be1b4d6595faf79b9e0abe" Aug 04 19:58:54.630147 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.630104 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:58:54.630593 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.630375 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" podUID="30e405a5-d80d-4de1-bfdc-88891e6be9de" containerName="maas-api" containerID="cri-o://bccf77065b07608b7efdfdff3f56cfcffb1a82f32dae36ae9c685b4164d785c5" gracePeriod=30 Aug 04 19:58:54.816951 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.816918 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq"] Aug 04 19:58:54.820095 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.820069 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4feefd-896vq"] Aug 04 19:58:54.849959 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.849931 2567 generic.go:356] "Generic (PLEG): container finished" podID="30e405a5-d80d-4de1-bfdc-88891e6be9de" containerID="bccf77065b07608b7efdfdff3f56cfcffb1a82f32dae36ae9c685b4164d785c5" exitCode=0 Aug 04 19:58:54.850117 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.849989 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" event={"ID":"30e405a5-d80d-4de1-bfdc-88891e6be9de","Type":"ContainerDied","Data":"bccf77065b07608b7efdfdff3f56cfcffb1a82f32dae36ae9c685b4164d785c5"} Aug 04 19:58:54.880202 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:54.880150 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:58:55.003281 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.003252 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6m26\" (UniqueName: \"kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26\") pod \"30e405a5-d80d-4de1-bfdc-88891e6be9de\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " Aug 04 19:58:55.003445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.003324 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca\") pod \"30e405a5-d80d-4de1-bfdc-88891e6be9de\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " Aug 04 19:58:55.003445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.003401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") pod \"30e405a5-d80d-4de1-bfdc-88891e6be9de\" (UID: \"30e405a5-d80d-4de1-bfdc-88891e6be9de\") " Aug 04 19:58:55.003698 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.003671 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30e405a5-d80d-4de1-bfdc-88891e6be9de" (UID: "30e405a5-d80d-4de1-bfdc-88891e6be9de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:58:55.005498 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.005477 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30e405a5-d80d-4de1-bfdc-88891e6be9de" (UID: "30e405a5-d80d-4de1-bfdc-88891e6be9de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:58:55.005560 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.005509 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26" (OuterVolumeSpecName: "kube-api-access-p6m26") pod "30e405a5-d80d-4de1-bfdc-88891e6be9de" (UID: "30e405a5-d80d-4de1-bfdc-88891e6be9de"). InnerVolumeSpecName "kube-api-access-p6m26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:58:55.104761 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.104729 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e405a5-d80d-4de1-bfdc-88891e6be9de-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.104761 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.104753 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30e405a5-d80d-4de1-bfdc-88891e6be9de-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.104761 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.104763 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6m26\" (UniqueName: \"kubernetes.io/projected/30e405a5-d80d-4de1-bfdc-88891e6be9de-kube-api-access-p6m26\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:58:55.504172 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.504133 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" path="/var/lib/kubelet/pods/7167f9c2-8a1b-4e11-92fe-5a6f22b3885e/volumes" Aug 04 19:58:55.857017 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.856990 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" Aug 04 19:58:55.857495 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.857031 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb" event={"ID":"30e405a5-d80d-4de1-bfdc-88891e6be9de","Type":"ContainerDied","Data":"ea41442bb4ed38a6fd1cdffa1116960749fde5655d79b5d24e5ec5b062c00208"} Aug 04 19:58:55.857495 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.857119 2567 scope.go:117] "RemoveContainer" containerID="bccf77065b07608b7efdfdff3f56cfcffb1a82f32dae36ae9c685b4164d785c5" Aug 04 19:58:55.875939 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.875917 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:58:55.881227 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:55.881204 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4feefd-5c7f846c6b-fspfb"] Aug 04 19:58:57.497692 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:57.497661 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:58:57.503924 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:58:57.503898 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e405a5-d80d-4de1-bfdc-88891e6be9de" path="/var/lib/kubelet/pods/30e405a5-d80d-4de1-bfdc-88891e6be9de/volumes" Aug 04 19:59:01.803078 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.802976 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt"] Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803346 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e405a5-d80d-4de1-bfdc-88891e6be9de" containerName="maas-api" Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803358 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e405a5-d80d-4de1-bfdc-88891e6be9de" containerName="maas-api" Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803371 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" containerName="revoke-keys" Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803377 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" containerName="revoke-keys" Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803437 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167f9c2-8a1b-4e11-92fe-5a6f22b3885e" containerName="revoke-keys" Aug 04 19:59:01.803445 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.803447 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e405a5-d80d-4de1-bfdc-88891e6be9de" containerName="maas-api" Aug 04 19:59:01.807934 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.807917 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:01.811390 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.811367 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 19:59:01.815380 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.815360 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt"] Aug 04 19:59:01.861365 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.861338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt\" (UID: \"68a19795-2d2c-4f14-8bac-510aef41473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:01.962696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.962666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt\" (UID: \"68a19795-2d2c-4f14-8bac-510aef41473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:01.963311 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:01.963290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt\" (UID: \"68a19795-2d2c-4f14-8bac-510aef41473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:02.119853 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:02.119767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:02.261291 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:02.261267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt"] Aug 04 19:59:02.262935 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:59:02.262907 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a19795_2d2c_4f14_8bac_510aef41473a.slice/crio-d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69 WatchSource:0}: Error finding container d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69: Status 404 returned error can't find the container with id d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69 Aug 04 19:59:02.898302 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:02.898260 2567 generic.go:356] "Generic (PLEG): container finished" podID="68a19795-2d2c-4f14-8bac-510aef41473a" containerID="4ce7e5a9ffc91adc4f2c60a85d905185f023a7fa585d5b98e68c283f8ef527f2" exitCode=0 Aug 04 19:59:02.898660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:02.898308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" event={"ID":"68a19795-2d2c-4f14-8bac-510aef41473a","Type":"ContainerDied","Data":"4ce7e5a9ffc91adc4f2c60a85d905185f023a7fa585d5b98e68c283f8ef527f2"} Aug 04 19:59:02.898660 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:02.898377 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" event={"ID":"68a19795-2d2c-4f14-8bac-510aef41473a","Type":"ContainerStarted","Data":"d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69"} Aug 04 19:59:04.034855 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.034832 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:04.182855 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.182768 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca\") pod \"68a19795-2d2c-4f14-8bac-510aef41473a\" (UID: \"68a19795-2d2c-4f14-8bac-510aef41473a\") " Aug 04 19:59:04.183112 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.183091 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68a19795-2d2c-4f14-8bac-510aef41473a" (UID: "68a19795-2d2c-4f14-8bac-510aef41473a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:04.283700 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.283670 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a19795-2d2c-4f14-8bac-510aef41473a-maas-api-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:59:04.909352 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.909316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" event={"ID":"68a19795-2d2c-4f14-8bac-510aef41473a","Type":"ContainerDied","Data":"d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69"} Aug 04 19:59:04.909352 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.909362 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ba1d526c3496f8933fff2e87b4fb2778980a24b6b26ab1f53cca801b291d69" Aug 04 19:59:04.909576 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:04.909329 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt" Aug 04 19:59:17.340165 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.340127 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:59:17.340618 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.340435 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" podUID="1572c659-09bb-4830-9f64-b9d948b6d55b" containerName="maas-api" containerID="cri-o://c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f" gracePeriod=30 Aug 04 19:59:17.539883 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.539836 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt"] Aug 04 19:59:17.541878 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.541850 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2e00c3-gzxnt"] Aug 04 19:59:17.593959 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.593902 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:59:17.692463 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.692435 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls\") pod \"1572c659-09bb-4830-9f64-b9d948b6d55b\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " Aug 04 19:59:17.692622 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.692495 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljzr6\" (UniqueName: \"kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6\") pod \"1572c659-09bb-4830-9f64-b9d948b6d55b\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " Aug 04 19:59:17.692622 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.692536 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca\") pod \"1572c659-09bb-4830-9f64-b9d948b6d55b\" (UID: \"1572c659-09bb-4830-9f64-b9d948b6d55b\") " Aug 04 19:59:17.693064 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.693012 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1572c659-09bb-4830-9f64-b9d948b6d55b" (UID: "1572c659-09bb-4830-9f64-b9d948b6d55b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:17.694592 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.694571 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1572c659-09bb-4830-9f64-b9d948b6d55b" (UID: "1572c659-09bb-4830-9f64-b9d948b6d55b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:59:17.694696 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.694609 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6" (OuterVolumeSpecName: "kube-api-access-ljzr6") pod "1572c659-09bb-4830-9f64-b9d948b6d55b" (UID: "1572c659-09bb-4830-9f64-b9d948b6d55b"). InnerVolumeSpecName "kube-api-access-ljzr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:17.793943 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.793903 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1572c659-09bb-4830-9f64-b9d948b6d55b-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.793943 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.793939 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzr6\" (UniqueName: \"kubernetes.io/projected/1572c659-09bb-4830-9f64-b9d948b6d55b-kube-api-access-ljzr6\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.793943 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.793949 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1572c659-09bb-4830-9f64-b9d948b6d55b-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 19:59:17.982341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.982253 2567 generic.go:356] "Generic (PLEG): container finished" podID="1572c659-09bb-4830-9f64-b9d948b6d55b" containerID="c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f" exitCode=0 Aug 04 19:59:17.982341 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.982299 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" Aug 04 19:59:17.982532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.982334 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" event={"ID":"1572c659-09bb-4830-9f64-b9d948b6d55b","Type":"ContainerDied","Data":"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f"} Aug 04 19:59:17.982532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.982386 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67" event={"ID":"1572c659-09bb-4830-9f64-b9d948b6d55b","Type":"ContainerDied","Data":"2aff7d73818329c6d0483f2d6eb1de951ace5710e300f3d8e5f90893df9de8dc"} Aug 04 19:59:17.982532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.982408 2567 scope.go:117] "RemoveContainer" containerID="c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f" Aug 04 19:59:17.991461 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.991446 2567 scope.go:117] "RemoveContainer" containerID="c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f" Aug 04 19:59:17.991722 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:17.991697 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f\": container with ID starting with c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f not found: ID does not exist" containerID="c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f" Aug 04 19:59:17.991838 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:17.991732 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f"} err="failed to get container status \"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f\": rpc error: code = NotFound desc = could not find container \"c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f\": container with ID starting with c55c774ec849d87572647063324ed8366c1673b4809eaa1226148b7c3d381f5f not found: ID does not exist" Aug 04 19:59:18.005802 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:18.005777 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:59:18.008668 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:18.008646 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e00c3-656b87845b-srr67"] Aug 04 19:59:19.505024 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:19.504988 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1572c659-09bb-4830-9f64-b9d948b6d55b" path="/var/lib/kubelet/pods/1572c659-09bb-4830-9f64-b9d948b6d55b/volumes" Aug 04 19:59:19.505415 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:19.505341 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a19795-2d2c-4f14-8bac-510aef41473a" path="/var/lib/kubelet/pods/68a19795-2d2c-4f14-8bac-510aef41473a/volumes" Aug 04 19:59:19.590016 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:19.589980 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:59:29.831654 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.831620 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832017 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1572c659-09bb-4830-9f64-b9d948b6d55b" containerName="maas-api" Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832031 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1572c659-09bb-4830-9f64-b9d948b6d55b" containerName="maas-api" Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832041 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a19795-2d2c-4f14-8bac-510aef41473a" containerName="revoke-keys" Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832058 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a19795-2d2c-4f14-8bac-510aef41473a" containerName="revoke-keys" Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832137 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1572c659-09bb-4830-9f64-b9d948b6d55b" containerName="maas-api" Aug 04 19:59:29.832296 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.832147 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a19795-2d2c-4f14-8bac-510aef41473a" containerName="revoke-keys" Aug 04 19:59:29.836420 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.836403 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:29.839728 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.839708 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 19:59:29.841112 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.841085 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7lh5k\"" Aug 04 19:59:29.846965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.846944 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 19:59:29.848365 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.848337 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-03ae76-serving-cert\"" Aug 04 19:59:29.852335 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.852313 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 19:59:29.901390 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.901358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9knbx\" (UniqueName: \"kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:29.901526 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.901403 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:29.901526 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:29.901516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.002685 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.002656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9knbx\" (UniqueName: \"kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.002848 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.002692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.002848 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.002747 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.002848 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:30.002839 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-serving-cert: secret "maas-api-e2e-shared-a-03ae76-serving-cert" not found Aug 04 19:59:30.003044 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:30.002898 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls podName:987ede6f-2ce2-4138-b32d-b6f9ec917cfb nodeName:}" failed. No retries permitted until 2026-08-04 19:59:30.502883809 +0000 UTC m=+2631.472910311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls") pod "maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" (UID: "987ede6f-2ce2-4138-b32d-b6f9ec917cfb") : secret "maas-api-e2e-shared-a-03ae76-serving-cert" not found Aug 04 19:59:30.003397 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.003376 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.013814 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.013789 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9knbx\" (UniqueName: \"kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.507612 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:30.507578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:30.507821 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:30.507755 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-serving-cert: secret "maas-api-e2e-shared-a-03ae76-serving-cert" not found Aug 04 19:59:30.507897 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:30.507847 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls podName:987ede6f-2ce2-4138-b32d-b6f9ec917cfb nodeName:}" failed. No retries permitted until 2026-08-04 19:59:31.507824773 +0000 UTC m=+2632.477851286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls") pod "maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" (UID: "987ede6f-2ce2-4138-b32d-b6f9ec917cfb") : secret "maas-api-e2e-shared-a-03ae76-serving-cert" not found Aug 04 19:59:31.516882 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:31.516798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:31.519266 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:31.519243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:31.656954 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:31.656918 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:31.785270 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:31.785243 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 19:59:31.787538 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:59:31.787507 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987ede6f_2ce2_4138_b32d_b6f9ec917cfb.slice/crio-1e677f55b12e7e353d9c228f1fef969da46caa1fe2899fbe182fbbfb4b3f26c9 WatchSource:0}: Error finding container 1e677f55b12e7e353d9c228f1fef969da46caa1fe2899fbe182fbbfb4b3f26c9: Status 404 returned error can't find the container with id 1e677f55b12e7e353d9c228f1fef969da46caa1fe2899fbe182fbbfb4b3f26c9 Aug 04 19:59:32.062664 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:32.062587 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" event={"ID":"987ede6f-2ce2-4138-b32d-b6f9ec917cfb","Type":"ContainerStarted","Data":"1e677f55b12e7e353d9c228f1fef969da46caa1fe2899fbe182fbbfb4b3f26c9"} Aug 04 19:59:32.298582 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:32.298551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 19:59:33.069585 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:33.069553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" event={"ID":"987ede6f-2ce2-4138-b32d-b6f9ec917cfb","Type":"ContainerStarted","Data":"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851"} Aug 04 19:59:33.069967 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:33.069658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:33.089661 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:33.089622 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" podStartSLOduration=3.691001834 podStartE2EDuration="4.089609914s" totalImagesPullingTime="398.60808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:59:33.08727081 +0000 UTC m=+2634.057297324" watchObservedRunningTime="2026-08-04 19:59:33.089609914 +0000 UTC m=+2634.059636433" Aug 04 19:59:39.080464 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:39.080436 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 19:59:39.822789 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:39.822759 2567 scope.go:117] "RemoveContainer" containerID="4b10e23c20bc628ceb9f4cfa6472aa9b7f5f186db76ad4183afdd8355a70843b" Aug 04 19:59:58.939242 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:58.939211 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 19:59:58.942532 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:58.942517 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:58.944896 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:58.944878 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-15a874-serving-cert\"" Aug 04 19:59:58.951379 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:58.951354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 19:59:59.052921 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.052889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.053089 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.052937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.053089 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.053071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2qh\" (UniqueName: \"kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.154028 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.153995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.154222 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.154083 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2qh\" (UniqueName: \"kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.154222 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.154116 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.154222 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:59.154212 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-serving-cert: secret "maas-api-e2e-shared-b-15a874-serving-cert" not found Aug 04 19:59:59.154415 ip-10-0-140-250 kubenswrapper[2567]: E0804 19:59:59.154269 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls podName:f0547b7e-fbd7-4e8f-8087-668c857ad71e nodeName:}" failed. No retries permitted until 2026-08-04 19:59:59.654251525 +0000 UTC m=+2660.624278022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls") pod "maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" (UID: "f0547b7e-fbd7-4e8f-8087-668c857ad71e") : secret "maas-api-e2e-shared-b-15a874-serving-cert" not found Aug 04 19:59:59.154715 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.154691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.162590 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.162562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2qh\" (UniqueName: \"kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.658965 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.658928 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.661343 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.661319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.854144 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.854106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 19:59:59.985765 ip-10-0-140-250 kubenswrapper[2567]: I0804 19:59:59.985740 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 19:59:59.987660 ip-10-0-140-250 kubenswrapper[2567]: W0804 19:59:59.987633 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0547b7e_fbd7_4e8f_8087_668c857ad71e.slice/crio-410e7cdf3d9c1fcf004c9f800d50ba5d2ae352f38c61f2ffa2106ec9725ad023 WatchSource:0}: Error finding container 410e7cdf3d9c1fcf004c9f800d50ba5d2ae352f38c61f2ffa2106ec9725ad023: Status 404 returned error can't find the container with id 410e7cdf3d9c1fcf004c9f800d50ba5d2ae352f38c61f2ffa2106ec9725ad023 Aug 04 20:00:00.130565 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.130529 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs"] Aug 04 20:00:00.135342 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.135320 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:00.135486 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.135456 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" Aug 04 20:00:00.138241 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.138224 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b"] Aug 04 20:00:00.138379 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.138360 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" Aug 04 20:00:00.139989 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.139972 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:00:00.141211 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.141198 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" Aug 04 20:00:00.146660 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.146638 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs"] Aug 04 20:00:00.158332 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.158306 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" Aug 04 20:00:00.172287 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.171098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" Aug 04 20:00:00.174638 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.174620 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:00.176875 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.176851 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b"] Aug 04 20:00:00.206876 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.206847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" Aug 04 20:00:00.224106 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.223790 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" event={"ID":"f0547b7e-fbd7-4e8f-8087-668c857ad71e","Type":"ContainerStarted","Data":"410e7cdf3d9c1fcf004c9f800d50ba5d2ae352f38c61f2ffa2106ec9725ad023"} Aug 04 20:00:00.344945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.344918 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs"] Aug 04 20:00:00.348187 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:00:00.348157 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3980cde5_0a59_4cd4_84bf_db590f40d427.slice/crio-66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877 WatchSource:0}: Error finding container 66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877: Status 404 returned error can't find the container with id 66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877 Aug 04 20:00:00.358007 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.357979 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:00.386546 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:00.386523 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b"] Aug 04 20:00:00.394105 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:00:00.394082 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ada90a_f489_466e_b605_7d684c30f74b.slice/crio-a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3 WatchSource:0}: Error finding container a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3: Status 404 returned error can't find the container with id a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3 Aug 04 20:00:01.232707 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.232664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" event={"ID":"d6ada90a-f489-466e-b605-7d684c30f74b","Type":"ContainerStarted","Data":"0519b969e1b61e049017be0c45450953c0c4a2f5e98bb6e1c53139b4308c8c0a"} Aug 04 20:00:01.233111 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.232744 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" event={"ID":"d6ada90a-f489-466e-b605-7d684c30f74b","Type":"ContainerStarted","Data":"a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3"} Aug 04 20:00:01.235275 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.235234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" event={"ID":"f0547b7e-fbd7-4e8f-8087-668c857ad71e","Type":"ContainerStarted","Data":"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf"} Aug 04 20:00:01.235405 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.235353 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 20:00:01.237039 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.237014 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerStarted","Data":"f9a344c2d844f320f0c2ff9bfc10fa1964ced2d46ddc4633471be3ac9cce6f7c"} Aug 04 20:00:01.237164 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.237079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerStarted","Data":"e882794866a1e37763b184b36863b42710ec2b53ab54eaa1ddc8be88b6105eab"} Aug 04 20:00:01.238723 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.238700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" event={"ID":"3980cde5-0a59-4cd4-84bf-db590f40d427","Type":"ContainerStarted","Data":"c37c0496a0e480188f7f271526eeaac76cf47ec8353f127038c418a523c390b2"} Aug 04 20:00:01.238867 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.238745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" event={"ID":"3980cde5-0a59-4cd4-84bf-db590f40d427","Type":"ContainerStarted","Data":"66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877"} Aug 04 20:00:01.248485 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.248448 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" podStartSLOduration=1.248435797 podStartE2EDuration="1.248435797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:01.24770607 +0000 UTC m=+2662.217732590" watchObservedRunningTime="2026-08-04 20:00:01.248435797 +0000 UTC m=+2662.218462293" Aug 04 20:00:01.263940 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.263836 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" podStartSLOduration=2.8841859899999998 podStartE2EDuration="3.26382113s" totalImagesPullingTime="379.63514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:01.262860531 +0000 UTC m=+2662.232887049" watchObservedRunningTime="2026-08-04 20:00:01.26382113 +0000 UTC m=+2662.233847650" Aug 04 20:00:01.277008 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.276972 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" podStartSLOduration=1.276961178 podStartE2EDuration="1.276961178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:01.276232252 +0000 UTC m=+2662.246258771" watchObservedRunningTime="2026-08-04 20:00:01.276961178 +0000 UTC m=+2662.246987675" Aug 04 20:00:01.290796 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.290762 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" podStartSLOduration=1.290749916 podStartE2EDuration="1.290749916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:01.290267042 +0000 UTC m=+2662.260293563" watchObservedRunningTime="2026-08-04 20:00:01.290749916 +0000 UTC m=+2662.260776435" Aug 04 20:00:01.511426 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:01.511351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:00:02.246475 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.246441 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerID="f9a344c2d844f320f0c2ff9bfc10fa1964ced2d46ddc4633471be3ac9cce6f7c" exitCode=7 Aug 04 20:00:02.246895 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.246511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerDied","Data":"f9a344c2d844f320f0c2ff9bfc10fa1964ced2d46ddc4633471be3ac9cce6f7c"} Aug 04 20:00:02.246895 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.246672 2567 scope.go:117] "RemoveContainer" containerID="f9a344c2d844f320f0c2ff9bfc10fa1964ced2d46ddc4633471be3ac9cce6f7c" Aug 04 20:00:02.248065 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.248033 2567 generic.go:356] "Generic (PLEG): container finished" podID="3980cde5-0a59-4cd4-84bf-db590f40d427" containerID="c37c0496a0e480188f7f271526eeaac76cf47ec8353f127038c418a523c390b2" exitCode=0 Aug 04 20:00:02.248161 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.248116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" event={"ID":"3980cde5-0a59-4cd4-84bf-db590f40d427","Type":"ContainerDied","Data":"c37c0496a0e480188f7f271526eeaac76cf47ec8353f127038c418a523c390b2"} Aug 04 20:00:02.249681 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.249663 2567 generic.go:356] "Generic (PLEG): container finished" podID="d6ada90a-f489-466e-b605-7d684c30f74b" containerID="0519b969e1b61e049017be0c45450953c0c4a2f5e98bb6e1c53139b4308c8c0a" exitCode=0 Aug 04 20:00:02.249763 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:02.249724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" event={"ID":"d6ada90a-f489-466e-b605-7d684c30f74b","Type":"ContainerDied","Data":"0519b969e1b61e049017be0c45450953c0c4a2f5e98bb6e1c53139b4308c8c0a"} Aug 04 20:00:03.260144 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.260030 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerID="cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519" exitCode=7 Aug 04 20:00:03.260144 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.260119 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerDied","Data":"cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519"} Aug 04 20:00:03.260812 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.260183 2567 scope.go:117] "RemoveContainer" containerID="f9a344c2d844f320f0c2ff9bfc10fa1964ced2d46ddc4633471be3ac9cce6f7c" Aug 04 20:00:03.260812 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.260335 2567 scope.go:117] "RemoveContainer" containerID="cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519" Aug 04 20:00:03.260812 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:00:03.260572 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g_odh-ai-gateway-infra(c4256943-42de-458e-8d1b-b6e8d9bed7a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" Aug 04 20:00:03.397386 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.397364 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" Aug 04 20:00:03.420682 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:03.420661 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" Aug 04 20:00:04.267498 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.267469 2567 scope.go:117] "RemoveContainer" containerID="cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519" Aug 04 20:00:04.267971 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:00:04.267689 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g_odh-ai-gateway-infra(c4256943-42de-458e-8d1b-b6e8d9bed7a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" Aug 04 20:00:04.268579 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.268559 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" Aug 04 20:00:04.268712 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.268598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764560-t9pgs" event={"ID":"3980cde5-0a59-4cd4-84bf-db590f40d427","Type":"ContainerDied","Data":"66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877"} Aug 04 20:00:04.268712 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.268636 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66eeb34a105654e00c6af98cc35f6bad5f554e2e63c118549dd1641e31c68877" Aug 04 20:00:04.269950 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.269926 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" event={"ID":"d6ada90a-f489-466e-b605-7d684c30f74b","Type":"ContainerDied","Data":"a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3"} Aug 04 20:00:04.270042 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.269964 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a738af7c269a855239cabb983b6d6ac5d0b9c2496069e5c8dc256c31322d07b3" Aug 04 20:00:04.270042 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:04.269933 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b" Aug 04 20:00:07.255404 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:07.255373 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 20:00:15.498519 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:15.498487 2567 scope.go:117] "RemoveContainer" containerID="cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519" Aug 04 20:00:16.348759 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:16.348709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerStarted","Data":"1aececcc8402c1a678a0fba604f51e195f651fecb4f26dc3b2a802417c531b0a"} Aug 04 20:00:17.356351 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:17.356315 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerID="1aececcc8402c1a678a0fba604f51e195f651fecb4f26dc3b2a802417c531b0a" exitCode=0 Aug 04 20:00:17.356806 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:17.356388 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerDied","Data":"1aececcc8402c1a678a0fba604f51e195f651fecb4f26dc3b2a802417c531b0a"} Aug 04 20:00:17.356806 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:17.356465 2567 scope.go:117] "RemoveContainer" containerID="cfa23cebdefc1d071615e780dfc33131dfdad779c23384534be18dc16d34f519" Aug 04 20:00:17.380255 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:17.380233 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:18.495598 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:18.495572 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" Aug 04 20:00:19.371211 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.371183 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" Aug 04 20:00:19.371402 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.371220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g" event={"ID":"c4256943-42de-458e-8d1b-b6e8d9bed7a0","Type":"ContainerDied","Data":"e882794866a1e37763b184b36863b42710ec2b53ab54eaa1ddc8be88b6105eab"} Aug 04 20:00:19.371402 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.371262 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e882794866a1e37763b184b36863b42710ec2b53ab54eaa1ddc8be88b6105eab" Aug 04 20:00:19.401163 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.401135 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:19.406415 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.406396 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-15a874-29764560-xss7g"] Aug 04 20:00:19.508203 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:19.505939 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" path="/var/lib/kubelet/pods/c4256943-42de-458e-8d1b-b6e8d9bed7a0/volumes" Aug 04 20:00:36.105729 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:00:36.105693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:01:24.722448 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:01:24.722414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:01:49.183726 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:01:49.183692 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:02:04.890317 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:02:04.890227 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:02:15.594454 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:02:15.594419 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:02:31.391266 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:02:31.391221 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:02:46.884977 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:02:46.884944 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:02:57.742522 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:02:57.742489 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:03:13.190889 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:03:13.190855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:03:37.299883 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:03:37.299801 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:03:39.877618 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:03:39.877590 2567 scope.go:117] "RemoveContainer" containerID="02a79452088bae389ad5a3b92edcae4ef899525181a9fd381fd1b60a46d74952" Aug 04 20:03:39.886137 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:03:39.886118 2567 scope.go:117] "RemoveContainer" containerID="0543af2ff66b7cb580bd3e6e5f740971aa697dd86f1b2545d579d1f3673216eb" Aug 04 20:03:48.294959 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:03:48.294912 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:04:03.711291 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:04:03.711258 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:04:18.988808 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:04:18.988770 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:04:29.797071 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:04:29.797021 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:04:45.525883 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:04:45.525853 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:05:00.886691 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:00.886649 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:05:12.396983 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:12.396946 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:05:27.089890 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:27.089855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:05:39.918816 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:39.918790 2567 scope.go:117] "RemoveContainer" containerID="4ce7e5a9ffc91adc4f2c60a85d905185f023a7fa585d5b98e68c283f8ef527f2" Aug 04 20:05:39.927167 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:39.927147 2567 scope.go:117] "RemoveContainer" containerID="f70473a9a77eebce6d8aeac6d1eece0e8f26a7be614dd3fe3dfc7e44e531d7ec" Aug 04 20:05:42.611946 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:42.611914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:05:52.495967 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:05:52.495929 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:06:07.293200 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:06:07.293166 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:06:39.949476 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:06:39.949400 2567 scope.go:117] "RemoveContainer" containerID="1aececcc8402c1a678a0fba604f51e195f651fecb4f26dc3b2a802417c531b0a" Aug 04 20:06:41.652702 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:06:41.652669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:07:15.790598 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:07:15.790564 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:07:27.698774 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:07:27.698739 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:07:49.084775 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:07:49.084744 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:08:00.397805 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:08:00.397769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:08:06.185928 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:08:06.185895 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:08:20.589435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:08:20.589398 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:08:56.590600 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:08:56.590561 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:09:04.690114 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:04.690073 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:09:36.484726 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:36.484640 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:09:43.510349 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510310 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880344496"] Aug 04 20:09:43.510861 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510844 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.510909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510866 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.510909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510878 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3980cde5-0a59-4cd4-84bf-db590f40d427" containerName="cleanup" Aug 04 20:09:43.510909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510884 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3980cde5-0a59-4cd4-84bf-db590f40d427" containerName="cleanup" Aug 04 20:09:43.510909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510897 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.510909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510902 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510914 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510919 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510934 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ada90a-f489-466e-b605-7d684c30f74b" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510940 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ada90a-f489-466e-b605-7d684c30f74b" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.510997 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.511005 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.511012 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ada90a-f489-466e-b605-7d684c30f74b" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.511017 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4256943-42de-458e-8d1b-b6e8d9bed7a0" containerName="cleanup" Aug 04 20:09:43.511086 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.511025 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3980cde5-0a59-4cd4-84bf-db590f40d427" containerName="cleanup" Aug 04 20:09:43.514017 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.514002 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:43.516710 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.516654 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nwqwj\"" Aug 04 20:09:43.519594 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.519575 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880344496"] Aug 04 20:09:43.571772 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.571739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vf27\" (UniqueName: \"kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27\") pod \"test-curl-7337-140024880344496\" (UID: \"6fbf208c-50f8-47a1-be15-626efa713459\") " pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:43.672531 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.672497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vf27\" (UniqueName: \"kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27\") pod \"test-curl-7337-140024880344496\" (UID: \"6fbf208c-50f8-47a1-be15-626efa713459\") " pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:43.680183 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.680151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vf27\" (UniqueName: \"kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27\") pod \"test-curl-7337-140024880344496\" (UID: \"6fbf208c-50f8-47a1-be15-626efa713459\") " pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:43.825367 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.825332 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:43.954002 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.953977 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880344496"] Aug 04 20:09:43.955393 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:43.955366 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fbf208c_50f8_47a1_be15_626efa713459.slice/crio-bccab3d39b6061d2dddf1c67472d1452fd651ec2571f28783d92ca14845da004 WatchSource:0}: Error finding container bccab3d39b6061d2dddf1c67472d1452fd651ec2571f28783d92ca14845da004: Status 404 returned error can't find the container with id bccab3d39b6061d2dddf1c67472d1452fd651ec2571f28783d92ca14845da004 Aug 04 20:09:43.956790 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:43.956770 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 20:09:44.436307 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:44.436272 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880344496" event={"ID":"6fbf208c-50f8-47a1-be15-626efa713459","Type":"ContainerStarted","Data":"bccab3d39b6061d2dddf1c67472d1452fd651ec2571f28783d92ca14845da004"} Aug 04 20:09:45.444558 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:45.444520 2567 generic.go:356] "Generic (PLEG): container finished" podID="6fbf208c-50f8-47a1-be15-626efa713459" containerID="a0060b8b26890cdc8a7452e5744ab4a45c25a1b9615b2929fde0ef69c47948e4" exitCode=0 Aug 04 20:09:45.444997 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:45.444610 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880344496" event={"ID":"6fbf208c-50f8-47a1-be15-626efa713459","Type":"ContainerDied","Data":"a0060b8b26890cdc8a7452e5744ab4a45c25a1b9615b2929fde0ef69c47948e4"} Aug 04 20:09:46.582694 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.582672 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:46.617589 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.617565 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880344496_6fbf208c-50f8-47a1-be15-626efa713459/test-curl-7337-140024880344496/0.log" Aug 04 20:09:46.638024 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.637986 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880344496"] Aug 04 20:09:46.641448 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.641426 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880344496"] Aug 04 20:09:46.697251 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.697216 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vf27\" (UniqueName: \"kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27\") pod \"6fbf208c-50f8-47a1-be15-626efa713459\" (UID: \"6fbf208c-50f8-47a1-be15-626efa713459\") " Aug 04 20:09:46.699387 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.699361 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27" (OuterVolumeSpecName: "kube-api-access-5vf27") pod "6fbf208c-50f8-47a1-be15-626efa713459" (UID: "6fbf208c-50f8-47a1-be15-626efa713459"). InnerVolumeSpecName "kube-api-access-5vf27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:46.762741 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.762708 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880344752"] Aug 04 20:09:46.763096 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.763082 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf208c-50f8-47a1-be15-626efa713459" containerName="test-curl-7337-140024880344496" Aug 04 20:09:46.763147 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.763098 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf208c-50f8-47a1-be15-626efa713459" containerName="test-curl-7337-140024880344496" Aug 04 20:09:46.763187 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.763161 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf208c-50f8-47a1-be15-626efa713459" containerName="test-curl-7337-140024880344496" Aug 04 20:09:46.766221 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.766206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:46.770105 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.770078 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880344752"] Aug 04 20:09:46.798580 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.798550 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vf27\" (UniqueName: \"kubernetes.io/projected/6fbf208c-50f8-47a1-be15-626efa713459-kube-api-access-5vf27\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:09:46.899420 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.899385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqggb\" (UniqueName: \"kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb\") pod \"test-curl-7337-140024880344752\" (UID: \"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f\") " pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:47.000107 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:46.999972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqggb\" (UniqueName: \"kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb\") pod \"test-curl-7337-140024880344752\" (UID: \"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f\") " pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:47.008924 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.008901 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqggb\" (UniqueName: \"kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb\") pod \"test-curl-7337-140024880344752\" (UID: \"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f\") " pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:47.076585 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.076554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:47.211229 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.211199 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880344752"] Aug 04 20:09:47.212553 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:47.212511 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b7bd1c7_4c99_43e8_a98f_3f45298cc05f.slice/crio-3717a906e50803c57e8c3b58ca6fb26e05f3a53be3b843d2950ebe5c4375855b WatchSource:0}: Error finding container 3717a906e50803c57e8c3b58ca6fb26e05f3a53be3b843d2950ebe5c4375855b: Status 404 returned error can't find the container with id 3717a906e50803c57e8c3b58ca6fb26e05f3a53be3b843d2950ebe5c4375855b Aug 04 20:09:47.457264 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.457229 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880344752" event={"ID":"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f","Type":"ContainerStarted","Data":"3717a906e50803c57e8c3b58ca6fb26e05f3a53be3b843d2950ebe5c4375855b"} Aug 04 20:09:47.458360 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.458344 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bccab3d39b6061d2dddf1c67472d1452fd651ec2571f28783d92ca14845da004" Aug 04 20:09:47.458360 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.458362 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:47.468857 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:09:47.468832 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-140024880344496\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="6fbf208c-50f8-47a1-be15-626efa713459" pod="opendatahub/test-curl-7337-140024880344496" Aug 04 20:09:47.504666 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.504635 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbf208c-50f8-47a1-be15-626efa713459" path="/var/lib/kubelet/pods/6fbf208c-50f8-47a1-be15-626efa713459/volumes" Aug 04 20:09:47.990404 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:47.990364 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:09:48.466274 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:48.466241 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" containerID="a4259896842ec7e935432fe657b529eb7c0bb3ba9cc8aaa7b600eb70793cc90a" exitCode=0 Aug 04 20:09:48.466445 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:48.466323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880344752" event={"ID":"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f","Type":"ContainerDied","Data":"a4259896842ec7e935432fe657b529eb7c0bb3ba9cc8aaa7b600eb70793cc90a"} Aug 04 20:09:49.611540 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.611517 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:49.633623 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.633599 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880344752_6b7bd1c7-4c99-43e8-a98f-3f45298cc05f/test-curl-7337-140024880344752/0.log" Aug 04 20:09:49.658521 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.658486 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880344752"] Aug 04 20:09:49.660599 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.660580 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880344752"] Aug 04 20:09:49.725860 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.725772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqggb\" (UniqueName: \"kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb\") pod \"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f\" (UID: \"6b7bd1c7-4c99-43e8-a98f-3f45298cc05f\") " Aug 04 20:09:49.727859 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.727831 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb" (OuterVolumeSpecName: "kube-api-access-nqggb") pod "6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" (UID: "6b7bd1c7-4c99-43e8-a98f-3f45298cc05f"). InnerVolumeSpecName "kube-api-access-nqggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:49.821924 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.821882 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:49.822382 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.822362 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" containerName="test-curl-7337-140024880344752" Aug 04 20:09:49.822475 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.822384 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" containerName="test-curl-7337-140024880344752" Aug 04 20:09:49.822546 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.822475 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" containerName="test-curl-7337-140024880344752" Aug 04 20:09:49.825412 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.825393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:49.826880 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.826858 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqggb\" (UniqueName: \"kubernetes.io/projected/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f-kube-api-access-nqggb\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:09:49.829774 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.829754 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:49.927955 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:49.927922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7rs\" (UniqueName: \"kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs\") pod \"test-curl-7337-140024880509120\" (UID: \"78707f31-45c5-4e79-8990-d1fab2c3c893\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:50.029096 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.028975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7rs\" (UniqueName: \"kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs\") pod \"test-curl-7337-140024880509120\" (UID: \"78707f31-45c5-4e79-8990-d1fab2c3c893\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:50.046617 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.046586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7rs\" (UniqueName: \"kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs\") pod \"test-curl-7337-140024880509120\" (UID: \"78707f31-45c5-4e79-8990-d1fab2c3c893\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:50.136687 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.136654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:50.263926 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.263896 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:50.265100 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:50.265077 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78707f31_45c5_4e79_8990_d1fab2c3c893.slice/crio-422cb1d8aefe5571ecc9e788528535c85331f4a11a434cd2ce4a855f5694d657 WatchSource:0}: Error finding container 422cb1d8aefe5571ecc9e788528535c85331f4a11a434cd2ce4a855f5694d657: Status 404 returned error can't find the container with id 422cb1d8aefe5571ecc9e788528535c85331f4a11a434cd2ce4a855f5694d657 Aug 04 20:09:50.477879 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.477835 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:50.477879 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.477885 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3717a906e50803c57e8c3b58ca6fb26e05f3a53be3b843d2950ebe5c4375855b" Aug 04 20:09:50.479003 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:50.478978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880509120" event={"ID":"78707f31-45c5-4e79-8990-d1fab2c3c893","Type":"ContainerStarted","Data":"422cb1d8aefe5571ecc9e788528535c85331f4a11a434cd2ce4a855f5694d657"} Aug 04 20:09:50.488571 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:09:50.488545 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-140024880344752\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:51.485522 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:51.485485 2567 generic.go:356] "Generic (PLEG): container finished" podID="78707f31-45c5-4e79-8990-d1fab2c3c893" containerID="b2c5cb90b5df2470a857f2fe100a6532b594a53e393f315a2dc52177e3c3c288" exitCode=0 Aug 04 20:09:51.485899 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:51.485532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880509120" event={"ID":"78707f31-45c5-4e79-8990-d1fab2c3c893","Type":"ContainerDied","Data":"b2c5cb90b5df2470a857f2fe100a6532b594a53e393f315a2dc52177e3c3c288"} Aug 04 20:09:51.487868 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:09:51.487844 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-140024880344752\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" pod="opendatahub/test-curl-7337-140024880344752" Aug 04 20:09:51.505162 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:51.505132 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7bd1c7-4c99-43e8-a98f-3f45298cc05f" path="/var/lib/kubelet/pods/6b7bd1c7-4c99-43e8-a98f-3f45298cc05f/volumes" Aug 04 20:09:52.620444 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.620418 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:52.640188 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.640163 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880509120_78707f31-45c5-4e79-8990-d1fab2c3c893/test-curl-7337-140024880509120/0.log" Aug 04 20:09:52.663495 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.663473 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:52.666749 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.666728 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:52.756557 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.756488 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj7rs\" (UniqueName: \"kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs\") pod \"78707f31-45c5-4e79-8990-d1fab2c3c893\" (UID: \"78707f31-45c5-4e79-8990-d1fab2c3c893\") " Aug 04 20:09:52.758521 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.758493 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs" (OuterVolumeSpecName: "kube-api-access-mj7rs") pod "78707f31-45c5-4e79-8990-d1fab2c3c893" (UID: "78707f31-45c5-4e79-8990-d1fab2c3c893"). InnerVolumeSpecName "kube-api-access-mj7rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:52.779010 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.778988 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:52.779348 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.779336 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78707f31-45c5-4e79-8990-d1fab2c3c893" containerName="test-curl-7337-140024880509120" Aug 04 20:09:52.779403 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.779350 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="78707f31-45c5-4e79-8990-d1fab2c3c893" containerName="test-curl-7337-140024880509120" Aug 04 20:09:52.779442 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.779419 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="78707f31-45c5-4e79-8990-d1fab2c3c893" containerName="test-curl-7337-140024880509120" Aug 04 20:09:52.782273 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.782259 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:52.788879 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.788850 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:52.858031 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.858002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4q5\" (UniqueName: \"kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5\") pod \"test-curl-7337-140024880509120\" (UID: \"30d684c7-58f4-47b9-a529-8debbdb27ebc\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:52.858208 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.858177 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7rs\" (UniqueName: \"kubernetes.io/projected/78707f31-45c5-4e79-8990-d1fab2c3c893-kube-api-access-mj7rs\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:09:52.959315 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.959281 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4q5\" (UniqueName: \"kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5\") pod \"test-curl-7337-140024880509120\" (UID: \"30d684c7-58f4-47b9-a529-8debbdb27ebc\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:52.969336 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:52.969300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4q5\" (UniqueName: \"kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5\") pod \"test-curl-7337-140024880509120\" (UID: \"30d684c7-58f4-47b9-a529-8debbdb27ebc\") " pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:53.092334 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.092301 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:53.222332 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.222309 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:53.223974 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:53.223944 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30d684c7_58f4_47b9_a529_8debbdb27ebc.slice/crio-bdd847876f2896427c35c58c73aa307018ce338ad20bbfd8653eb3055e99e60c WatchSource:0}: Error finding container bdd847876f2896427c35c58c73aa307018ce338ad20bbfd8653eb3055e99e60c: Status 404 returned error can't find the container with id bdd847876f2896427c35c58c73aa307018ce338ad20bbfd8653eb3055e99e60c Aug 04 20:09:53.498142 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.498034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880509120" event={"ID":"30d684c7-58f4-47b9-a529-8debbdb27ebc","Type":"ContainerStarted","Data":"bdd847876f2896427c35c58c73aa307018ce338ad20bbfd8653eb3055e99e60c"} Aug 04 20:09:53.499246 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.499227 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:53.504333 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.504309 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78707f31-45c5-4e79-8990-d1fab2c3c893" path="/var/lib/kubelet/pods/78707f31-45c5-4e79-8990-d1fab2c3c893/volumes" Aug 04 20:09:53.504660 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:53.504647 2567 scope.go:117] "RemoveContainer" containerID="b2c5cb90b5df2470a857f2fe100a6532b594a53e393f315a2dc52177e3c3c288" Aug 04 20:09:54.506651 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:54.506613 2567 generic.go:356] "Generic (PLEG): container finished" podID="30d684c7-58f4-47b9-a529-8debbdb27ebc" containerID="dc54dde2dcc382d83b5cd38d2c60b3f6b42be5c7b6653eacee0035229178873d" exitCode=0 Aug 04 20:09:54.507038 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:54.506700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880509120" event={"ID":"30d684c7-58f4-47b9-a529-8debbdb27ebc","Type":"ContainerDied","Data":"dc54dde2dcc382d83b5cd38d2c60b3f6b42be5c7b6653eacee0035229178873d"} Aug 04 20:09:55.646441 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.646411 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:55.665624 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.665599 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880509120_30d684c7-58f4-47b9-a529-8debbdb27ebc/test-curl-7337-140024880509120/0.log" Aug 04 20:09:55.689989 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.689961 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:55.693644 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.693624 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880509120"] Aug 04 20:09:55.785738 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.785720 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr4q5\" (UniqueName: \"kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5\") pod \"30d684c7-58f4-47b9-a529-8debbdb27ebc\" (UID: \"30d684c7-58f4-47b9-a529-8debbdb27ebc\") " Aug 04 20:09:55.787813 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.787790 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5" (OuterVolumeSpecName: "kube-api-access-lr4q5") pod "30d684c7-58f4-47b9-a529-8debbdb27ebc" (UID: "30d684c7-58f4-47b9-a529-8debbdb27ebc"). InnerVolumeSpecName "kube-api-access-lr4q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:55.809861 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.809836 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:55.810221 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.810208 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d684c7-58f4-47b9-a529-8debbdb27ebc" containerName="test-curl-7337-140024880509120" Aug 04 20:09:55.810278 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.810222 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d684c7-58f4-47b9-a529-8debbdb27ebc" containerName="test-curl-7337-140024880509120" Aug 04 20:09:55.810317 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.810297 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d684c7-58f4-47b9-a529-8debbdb27ebc" containerName="test-curl-7337-140024880509120" Aug 04 20:09:55.813220 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.813207 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:55.820546 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.820139 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:55.886797 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.886759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgz7\" (UniqueName: \"kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7\") pod \"test-curl-7337-140024880510128\" (UID: \"9978296f-14bd-4e38-be79-5f2ce1a2409c\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:55.886976 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.886864 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4q5\" (UniqueName: \"kubernetes.io/projected/30d684c7-58f4-47b9-a529-8debbdb27ebc-kube-api-access-lr4q5\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:09:55.988147 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.988109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgz7\" (UniqueName: \"kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7\") pod \"test-curl-7337-140024880510128\" (UID: \"9978296f-14bd-4e38-be79-5f2ce1a2409c\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:55.996895 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:55.996872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgz7\" (UniqueName: \"kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7\") pod \"test-curl-7337-140024880510128\" (UID: \"9978296f-14bd-4e38-be79-5f2ce1a2409c\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:56.122926 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:56.122838 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:56.249837 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:56.249811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:56.251089 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:56.251042 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9978296f_14bd_4e38_be79_5f2ce1a2409c.slice/crio-9f542e10cbe79ff7f6cb9c8f7e4e4e94e1e03b90504446e1aeb2e32c595a1554 WatchSource:0}: Error finding container 9f542e10cbe79ff7f6cb9c8f7e4e4e94e1e03b90504446e1aeb2e32c595a1554: Status 404 returned error can't find the container with id 9f542e10cbe79ff7f6cb9c8f7e4e4e94e1e03b90504446e1aeb2e32c595a1554 Aug 04 20:09:56.520887 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:56.520812 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880510128" event={"ID":"9978296f-14bd-4e38-be79-5f2ce1a2409c","Type":"ContainerStarted","Data":"9f542e10cbe79ff7f6cb9c8f7e4e4e94e1e03b90504446e1aeb2e32c595a1554"} Aug 04 20:09:56.522006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:56.521988 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd847876f2896427c35c58c73aa307018ce338ad20bbfd8653eb3055e99e60c" Aug 04 20:09:56.522006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:56.522002 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:56.532790 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:09:56.532768 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-140024880509120\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="30d684c7-58f4-47b9-a529-8debbdb27ebc" pod="opendatahub/test-curl-7337-140024880509120" Aug 04 20:09:57.505540 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:57.505506 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d684c7-58f4-47b9-a529-8debbdb27ebc" path="/var/lib/kubelet/pods/30d684c7-58f4-47b9-a529-8debbdb27ebc/volumes" Aug 04 20:09:57.528745 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:57.528716 2567 generic.go:356] "Generic (PLEG): container finished" podID="9978296f-14bd-4e38-be79-5f2ce1a2409c" containerID="cc6bd19d3f17f7b3eebff8edc870d42baebb4f4043251c98a06845d25addb16c" exitCode=0 Aug 04 20:09:57.528922 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:57.528803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880510128" event={"ID":"9978296f-14bd-4e38-be79-5f2ce1a2409c","Type":"ContainerDied","Data":"cc6bd19d3f17f7b3eebff8edc870d42baebb4f4043251c98a06845d25addb16c"} Aug 04 20:09:58.662480 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.662458 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:58.681477 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.681453 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880510128_9978296f-14bd-4e38-be79-5f2ce1a2409c/test-curl-7337-140024880510128/0.log" Aug 04 20:09:58.703754 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.703728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:58.709363 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.709343 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:58.812978 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.812906 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sgz7\" (UniqueName: \"kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7\") pod \"9978296f-14bd-4e38-be79-5f2ce1a2409c\" (UID: \"9978296f-14bd-4e38-be79-5f2ce1a2409c\") " Aug 04 20:09:58.814948 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.814926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7" (OuterVolumeSpecName: "kube-api-access-9sgz7") pod "9978296f-14bd-4e38-be79-5f2ce1a2409c" (UID: "9978296f-14bd-4e38-be79-5f2ce1a2409c"). InnerVolumeSpecName "kube-api-access-9sgz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:58.825546 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.825527 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:58.825897 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.825885 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9978296f-14bd-4e38-be79-5f2ce1a2409c" containerName="test-curl-7337-140024880510128" Aug 04 20:09:58.825945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.825898 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9978296f-14bd-4e38-be79-5f2ce1a2409c" containerName="test-curl-7337-140024880510128" Aug 04 20:09:58.825984 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.825956 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9978296f-14bd-4e38-be79-5f2ce1a2409c" containerName="test-curl-7337-140024880510128" Aug 04 20:09:58.829003 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.828990 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:58.837093 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.837073 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:58.914038 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.914009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dr68\" (UniqueName: \"kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68\") pod \"test-curl-7337-140024880510128\" (UID: \"ea4e8698-dd5a-47cb-a012-916f89e975d5\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:58.914182 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:58.914142 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sgz7\" (UniqueName: \"kubernetes.io/projected/9978296f-14bd-4e38-be79-5f2ce1a2409c-kube-api-access-9sgz7\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:09:59.015111 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.015079 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dr68\" (UniqueName: \"kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68\") pod \"test-curl-7337-140024880510128\" (UID: \"ea4e8698-dd5a-47cb-a012-916f89e975d5\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:59.023713 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.023692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dr68\" (UniqueName: \"kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68\") pod \"test-curl-7337-140024880510128\" (UID: \"ea4e8698-dd5a-47cb-a012-916f89e975d5\") " pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:59.139843 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.139756 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:59.273637 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.273607 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:09:59.275398 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:09:59.275373 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea4e8698_dd5a_47cb_a012_916f89e975d5.slice/crio-3abec7401dc42a9a297c8f6c35ebb5fa1c57be0e2695840fb461d1cbccb4d7bb WatchSource:0}: Error finding container 3abec7401dc42a9a297c8f6c35ebb5fa1c57be0e2695840fb461d1cbccb4d7bb: Status 404 returned error can't find the container with id 3abec7401dc42a9a297c8f6c35ebb5fa1c57be0e2695840fb461d1cbccb4d7bb Aug 04 20:09:59.504724 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.504647 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9978296f-14bd-4e38-be79-5f2ce1a2409c" path="/var/lib/kubelet/pods/9978296f-14bd-4e38-be79-5f2ce1a2409c/volumes" Aug 04 20:09:59.539570 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.539543 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:09:59.539741 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.539580 2567 scope.go:117] "RemoveContainer" containerID="cc6bd19d3f17f7b3eebff8edc870d42baebb4f4043251c98a06845d25addb16c" Aug 04 20:09:59.540856 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:09:59.540830 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880510128" event={"ID":"ea4e8698-dd5a-47cb-a012-916f89e975d5","Type":"ContainerStarted","Data":"3abec7401dc42a9a297c8f6c35ebb5fa1c57be0e2695840fb461d1cbccb4d7bb"} Aug 04 20:10:00.547554 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:00.547519 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea4e8698-dd5a-47cb-a012-916f89e975d5" containerID="e750587ebdf01d91904be93978c9dadc49c064e99c04a2149e1a7dc5c5ffedcf" exitCode=0 Aug 04 20:10:00.547983 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:00.547604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-140024880510128" event={"ID":"ea4e8698-dd5a-47cb-a012-916f89e975d5","Type":"ContainerDied","Data":"e750587ebdf01d91904be93978c9dadc49c064e99c04a2149e1a7dc5c5ffedcf"} Aug 04 20:10:01.692195 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.692169 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:10:01.711511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.711476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7337-140024880510128_ea4e8698-dd5a-47cb-a012-916f89e975d5/test-curl-7337-140024880510128/0.log" Aug 04 20:10:01.734595 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.734561 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:10:01.739379 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.739343 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-140024880510128"] Aug 04 20:10:01.842025 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.841933 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dr68\" (UniqueName: \"kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68\") pod \"ea4e8698-dd5a-47cb-a012-916f89e975d5\" (UID: \"ea4e8698-dd5a-47cb-a012-916f89e975d5\") " Aug 04 20:10:01.844039 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.844006 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68" (OuterVolumeSpecName: "kube-api-access-8dr68") pod "ea4e8698-dd5a-47cb-a012-916f89e975d5" (UID: "ea4e8698-dd5a-47cb-a012-916f89e975d5"). InnerVolumeSpecName "kube-api-access-8dr68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:10:01.942898 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:01.942860 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dr68\" (UniqueName: \"kubernetes.io/projected/ea4e8698-dd5a-47cb-a012-916f89e975d5-kube-api-access-8dr68\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:10:02.508822 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.508787 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:10:02.509256 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.509237 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4e8698-dd5a-47cb-a012-916f89e975d5" containerName="test-curl-7337-140024880510128" Aug 04 20:10:02.509256 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.509255 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4e8698-dd5a-47cb-a012-916f89e975d5" containerName="test-curl-7337-140024880510128" Aug 04 20:10:02.509403 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.509330 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4e8698-dd5a-47cb-a012-916f89e975d5" containerName="test-curl-7337-140024880510128" Aug 04 20:10:02.511327 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.511310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.514024 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.514005 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-41ab60-openshift-default-dockercfg-tzb4c\"" Aug 04 20:10:02.528828 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.528724 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:10:02.561092 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.561068 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3abec7401dc42a9a297c8f6c35ebb5fa1c57be0e2695840fb461d1cbccb4d7bb" Aug 04 20:10:02.561260 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.561097 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:10:02.571066 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:10:02.571025 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-140024880510128\" is forbidden: User \"system:node:ip-10-0-140-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-250.ec2.internal' and this object" podUID="ea4e8698-dd5a-47cb-a012-916f89e975d5" pod="opendatahub/test-curl-7337-140024880510128" Aug 04 20:10:02.649930 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.649895 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.649930 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.649932 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650162 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.649961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650162 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650025 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650162 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650272 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650272 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650272 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jll\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.650369 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.650285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.751784 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.751746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.751784 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.751783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.751799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.751840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jll\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.751880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752253 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752758 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.752878 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.752861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.754273 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.754243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.754417 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.754401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.759697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.759630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.759844 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.759825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jll\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll\") pod \"e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.829922 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.829882 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:02.971591 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:02.971565 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:10:02.973294 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:10:02.973262 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b26fe7d_c0af_4282_85ce_36002354ef44.slice/crio-0af0b423b0fad948f1f657d6fbbf592ec22bcde0c25ecfe255c7377a1dd1acec WatchSource:0}: Error finding container 0af0b423b0fad948f1f657d6fbbf592ec22bcde0c25ecfe255c7377a1dd1acec: Status 404 returned error can't find the container with id 0af0b423b0fad948f1f657d6fbbf592ec22bcde0c25ecfe255c7377a1dd1acec Aug 04 20:10:03.504480 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:03.504448 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4e8698-dd5a-47cb-a012-916f89e975d5" path="/var/lib/kubelet/pods/ea4e8698-dd5a-47cb-a012-916f89e975d5/volumes" Aug 04 20:10:03.567603 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:03.567576 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerStarted","Data":"0af0b423b0fad948f1f657d6fbbf592ec22bcde0c25ecfe255c7377a1dd1acec"} Aug 04 20:10:06.290711 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.290678 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:06.290981 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.290740 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:06.290981 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.290773 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:06.587143 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.587045 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerStarted","Data":"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605"} Aug 04 20:10:06.609418 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.609356 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podStartSLOduration=1.294420597 podStartE2EDuration="4.609340893s" totalImagesPullingTime="3.314920296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:06.607594071 +0000 UTC m=+3267.577620625" watchObservedRunningTime="2026-08-04 20:10:06.609340893 +0000 UTC m=+3267.579367470" Aug 04 20:10:06.830255 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:06.830219 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:07.836082 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:07.836034 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:08.597851 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:08.597820 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:08.598812 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:08.598793 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:09.603998 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.603970 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:10:09.606491 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.606475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.609173 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.609150 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-41ab60-serving-cert\"" Aug 04 20:10:09.618889 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.618867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:10:09.716633 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.716601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.716812 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.716684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmp2\" (UniqueName: \"kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.716812 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.716760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.817710 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.817681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.817871 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.817726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmp2\" (UniqueName: \"kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.817871 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.817771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.817960 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:10:09.817883 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-serving-cert: secret "maas-api-e2e-ipp-a-41ab60-serving-cert" not found Aug 04 20:10:09.817960 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:10:09.817943 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls podName:2359e5b1-5895-486e-af5b-d9b482d50d18 nodeName:}" failed. No retries permitted until 2026-08-04 20:10:10.317927058 +0000 UTC m=+3271.287953555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls") pod "maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" (UID: "2359e5b1-5895-486e-af5b-d9b482d50d18") : secret "maas-api-e2e-ipp-a-41ab60-serving-cert" not found Aug 04 20:10:09.818289 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.818271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:09.827225 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:09.827201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmp2\" (UniqueName: \"kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:10.322395 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:10.322360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:10.322592 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:10:10.322524 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-serving-cert: secret "maas-api-e2e-ipp-a-41ab60-serving-cert" not found Aug 04 20:10:10.322669 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:10:10.322598 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls podName:2359e5b1-5895-486e-af5b-d9b482d50d18 nodeName:}" failed. No retries permitted until 2026-08-04 20:10:11.322578493 +0000 UTC m=+3272.292605009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls") pod "maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" (UID: "2359e5b1-5895-486e-af5b-d9b482d50d18") : secret "maas-api-e2e-ipp-a-41ab60-serving-cert" not found Aug 04 20:10:11.331807 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:11.331753 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:11.334213 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:11.334192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:11.417645 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:11.417613 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:11.547836 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:11.547810 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:10:11.548934 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:10:11.548907 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2359e5b1_5895_486e_af5b_d9b482d50d18.slice/crio-646ba7fc87954be0dc1de0a9f1f37f34bd06c053e2785ec31ee3dcd23ecb95ec WatchSource:0}: Error finding container 646ba7fc87954be0dc1de0a9f1f37f34bd06c053e2785ec31ee3dcd23ecb95ec: Status 404 returned error can't find the container with id 646ba7fc87954be0dc1de0a9f1f37f34bd06c053e2785ec31ee3dcd23ecb95ec Aug 04 20:10:11.617121 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:11.617033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" event={"ID":"2359e5b1-5895-486e-af5b-d9b482d50d18","Type":"ContainerStarted","Data":"646ba7fc87954be0dc1de0a9f1f37f34bd06c053e2785ec31ee3dcd23ecb95ec"} Aug 04 20:10:12.624497 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:12.624457 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" event={"ID":"2359e5b1-5895-486e-af5b-d9b482d50d18","Type":"ContainerStarted","Data":"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b"} Aug 04 20:10:12.624937 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:12.624515 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:12.647010 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:12.646964 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" podStartSLOduration=3.139885313 podStartE2EDuration="3.646949216s" totalImagesPullingTime="507.063903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:12.64586208 +0000 UTC m=+3273.615888623" watchObservedRunningTime="2026-08-04 20:10:12.646949216 +0000 UTC m=+3273.616975742" Aug 04 20:10:13.887760 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:13.887730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:10:18.640864 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:18.640831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:10:23.599741 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.599671 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:23.600173 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.599731 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:23.693667 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.693597 2567 generic.go:356] "Generic (PLEG): container finished" podID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerID="10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605" exitCode=0 Aug 04 20:10:23.693795 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.693667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerDied","Data":"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605"} Aug 04 20:10:23.694022 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.694008 2567 scope.go:117] "RemoveContainer" containerID="10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605" Aug 04 20:10:23.695040 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.695013 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:23.695144 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.695102 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:23.695144 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:23.695134 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:24.701105 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:24.701042 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerStarted","Data":"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29"} Aug 04 20:10:24.831216 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:24.830761 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:24.831216 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:24.830889 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:24.831216 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:24.830932 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:25.830518 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:25.830480 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:25.831463 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:25.831075 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:26.831467 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:26.831037 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:26.831467 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:26.831115 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:27.831250 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:27.831216 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:27.831521 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:27.831495 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:28.831231 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:28.831194 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:28.831563 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:28.831519 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:29.830907 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:29.830871 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Aug 04 20:10:29.831095 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:29.830924 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Aug 04 20:10:30.833836 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:30.833807 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:30.834210 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:30.834078 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:30.834769 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:30.834751 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:10:34.486556 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.486526 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:10:34.489341 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.489323 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.492644 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.492617 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9cd851-openshift-default-dockercfg-d7p7z\"" Aug 04 20:10:34.504757 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.504733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:10:34.538661 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538818 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbxh\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538818 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538730 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538818 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538818 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538809 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538994 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538994 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.538994 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.538990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.539137 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.539015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640044 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640009 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640223 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640223 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640223 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640223 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640223 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640465 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640465 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640712 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640675 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640846 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640923 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbxh\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.640923 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640878 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.641025 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.640994 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.641320 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.641295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.648840 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.643635 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.648840 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.644230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.650677 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.650651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.650972 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.650956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbxh\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh\") pod \"e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.800399 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.800364 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:34.932441 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.932412 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:10:34.933478 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:10:34.933447 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabfdfc6_3c4c_47cd_aa72_0308618be1f6.slice/crio-f684e9c4d353d47ecc5993bfada4ff987cc6f729ef520db7107d89acd85620cc WatchSource:0}: Error finding container f684e9c4d353d47ecc5993bfada4ff987cc6f729ef520db7107d89acd85620cc: Status 404 returned error can't find the container with id f684e9c4d353d47ecc5993bfada4ff987cc6f729ef520db7107d89acd85620cc Aug 04 20:10:34.935735 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.935690 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:34.935842 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.935763 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:34.935842 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:34.935800 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:35.765365 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:35.765329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerStarted","Data":"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e"} Aug 04 20:10:35.765821 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:35.765383 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerStarted","Data":"f684e9c4d353d47ecc5993bfada4ff987cc6f729ef520db7107d89acd85620cc"} Aug 04 20:10:35.801353 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:35.801322 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:35.806385 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:35.806361 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:35.827364 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:35.827312 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podStartSLOduration=1.827296124 podStartE2EDuration="1.827296124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:35.789276199 +0000 UTC m=+3296.759302719" watchObservedRunningTime="2026-08-04 20:10:35.827296124 +0000 UTC m=+3296.797322642" Aug 04 20:10:36.771121 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:36.771084 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:36.772009 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:36.771992 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:41.490257 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:41.490228 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:10:51.772494 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.772458 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 20:10:51.772929 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.772522 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 20:10:51.857659 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.857624 2567 generic.go:356] "Generic (PLEG): container finished" podID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerID="22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e" exitCode=0 Aug 04 20:10:51.857816 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.857697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerDied","Data":"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e"} Aug 04 20:10:51.858109 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.858094 2567 scope.go:117] "RemoveContainer" containerID="22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e" Aug 04 20:10:51.859242 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.859212 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:51.859349 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.859292 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:51.859349 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:51.859338 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:10:52.865061 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:52.865024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerStarted","Data":"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e"} Aug 04 20:10:53.801196 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:53.801166 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:53.801375 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:53.801334 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 20:10:53.801423 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:53.801381 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 20:10:54.800887 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:54.800852 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 20:10:54.801298 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:54.800906 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 20:10:55.801447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:55.801408 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 20:10:55.801832 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:55.801464 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 20:10:56.800708 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:56.800675 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Aug 04 20:10:56.800875 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:56.800726 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Aug 04 20:10:57.805339 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:57.805312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:57.805788 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:57.805572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:10:57.806227 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:10:57.806212 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:11:29.294258 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:11:29.294222 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:11:59.091893 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:11:59.091855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:12:10.726490 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.726450 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:12:10.730416 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.730391 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.733102 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.733079 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8f4e87-openshift-default-dockercfg-jqt89\"" Aug 04 20:12:10.742377 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.742352 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:12:10.819684 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.819627 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820100 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820291 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820442 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820568 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820701 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820815 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820802 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.820942 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.820929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.821126 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.821110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922075 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922350 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922460 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922460 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922573 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922492 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922573 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922533 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922687 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922687 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.922687 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.922663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.923160 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.923124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.923374 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.923348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.923496 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.923389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.923496 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.923429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.924006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.923988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.925193 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.925173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.925301 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.925282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.939402 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.939378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:10.939512 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:10.939475 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f\") pod \"e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:11.043825 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.043794 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:11.186405 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.186378 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:12:11.187893 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:12:11.187869 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1cb84a_0e8b_4aae_9dbe_4f2e0357bfff.slice/crio-1ec2efd44acfb72dc70cf2405da3e8fa449f14343694950fd7caeeafd352d38f WatchSource:0}: Error finding container 1ec2efd44acfb72dc70cf2405da3e8fa449f14343694950fd7caeeafd352d38f: Status 404 returned error can't find the container with id 1ec2efd44acfb72dc70cf2405da3e8fa449f14343694950fd7caeeafd352d38f Aug 04 20:12:11.190012 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.189982 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:11.190087 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.190077 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:11.190125 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.190109 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:11.306570 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.306491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerStarted","Data":"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344"} Aug 04 20:12:11.306570 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.306548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerStarted","Data":"1ec2efd44acfb72dc70cf2405da3e8fa449f14343694950fd7caeeafd352d38f"} Aug 04 20:12:11.328914 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:11.328849 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podStartSLOduration=1.328827656 podStartE2EDuration="1.328827656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:12:11.324467149 +0000 UTC m=+3392.294493691" watchObservedRunningTime="2026-08-04 20:12:11.328827656 +0000 UTC m=+3392.298854176" Aug 04 20:12:12.044770 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:12.044732 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:12.049833 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:12.049808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:12.313674 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:12.313590 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:12.314434 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:12.314418 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:14.426533 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.426504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:14.430412 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.430390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.434780 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.434755 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8f4e87-serving-cert\"" Aug 04 20:12:14.447899 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.447880 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:14.566692 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.561089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.566692 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.561165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.566692 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.561207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8xx\" (UniqueName: \"kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.662453 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.662421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.662675 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.662486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.662675 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:12:14.662586 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-serving-cert: secret "maas-api-e2e-ipp-cleanup-8f4e87-serving-cert" not found Aug 04 20:12:14.662675 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:12:14.662668 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls podName:cc2c9d0b-f3a0-4d9f-8f86-00915674ab46 nodeName:}" failed. No retries permitted until 2026-08-04 20:12:15.162644468 +0000 UTC m=+3396.132670985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" (UID: "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46") : secret "maas-api-e2e-ipp-cleanup-8f4e87-serving-cert" not found Aug 04 20:12:14.662858 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.662694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8xx\" (UniqueName: \"kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.663292 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.663267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:14.676432 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:14.676407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8xx\" (UniqueName: \"kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:15.169515 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:15.169483 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:15.172095 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:15.172069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:15.341293 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:15.341259 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:15.478881 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:15.478852 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:15.479213 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:12:15.478900 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2c9d0b_f3a0_4d9f_8f86_00915674ab46.slice/crio-8011c3de4acbeb3457c31be2d10258521a6289777eb220c115d54bc2230d0211 WatchSource:0}: Error finding container 8011c3de4acbeb3457c31be2d10258521a6289777eb220c115d54bc2230d0211: Status 404 returned error can't find the container with id 8011c3de4acbeb3457c31be2d10258521a6289777eb220c115d54bc2230d0211 Aug 04 20:12:16.338396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:16.338307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" event={"ID":"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46","Type":"ContainerStarted","Data":"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3"} Aug 04 20:12:16.338396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:16.338368 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:16.338396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:16.338387 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" event={"ID":"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46","Type":"ContainerStarted","Data":"8011c3de4acbeb3457c31be2d10258521a6289777eb220c115d54bc2230d0211"} Aug 04 20:12:16.356251 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:16.356187 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" podStartSLOduration=1.797818914 podStartE2EDuration="2.356173244s" totalImagesPullingTime="558.35433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:12:16.355666592 +0000 UTC m=+3397.325693112" watchObservedRunningTime="2026-08-04 20:12:16.356173244 +0000 UTC m=+3397.326199780" Aug 04 20:12:19.601351 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:19.601318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:12:22.355851 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:22.355824 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:28.355210 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:28.355165 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 20:12:28.355589 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:28.355237 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 20:12:29.422961 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.422930 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerID="109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344" exitCode=0 Aug 04 20:12:29.423396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.423001 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerDied","Data":"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344"} Aug 04 20:12:29.423396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.423347 2567 scope.go:117] "RemoveContainer" containerID="109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344" Aug 04 20:12:29.424351 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.424325 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:29.424408 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.424386 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:29.424448 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:29.424420 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:12:30.431084 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:30.430665 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerStarted","Data":"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967"} Aug 04 20:12:31.045145 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:31.045107 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:31.045348 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:31.045228 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 20:12:31.045348 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:31.045279 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 20:12:32.045226 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:32.045184 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 20:12:32.045714 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:32.045259 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 20:12:33.044453 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:33.044414 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 20:12:33.044745 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:33.044712 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 20:12:34.045199 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:34.045156 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 20:12:34.045720 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:34.045223 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 20:12:35.044789 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:35.044742 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Aug 04 20:12:35.044998 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:35.044826 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Aug 04 20:12:36.048100 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:36.048071 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:36.048553 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:36.048470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:36.049037 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:36.049018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:12:55.053968 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.053933 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:55.054559 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.054193 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" podUID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" containerName="maas-api" containerID="cri-o://3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3" gracePeriod=30 Aug 04 20:12:55.310798 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.310707 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:55.446871 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.446843 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj8xx\" (UniqueName: \"kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx\") pod \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " Aug 04 20:12:55.447069 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.446916 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") pod \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " Aug 04 20:12:55.447069 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.446976 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca\") pod \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\" (UID: \"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46\") " Aug 04 20:12:55.447362 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.447340 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" (UID: "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:12:55.448944 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.448917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx" (OuterVolumeSpecName: "kube-api-access-lj8xx") pod "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" (UID: "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46"). InnerVolumeSpecName "kube-api-access-lj8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:12:55.449085 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.448999 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" (UID: "cc2c9d0b-f3a0-4d9f-8f86-00915674ab46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:12:55.548434 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.548404 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.548434 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.548435 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.548605 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.548449 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8xx\" (UniqueName: \"kubernetes.io/projected/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46-kube-api-access-lj8xx\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.581304 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.581242 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" containerID="3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3" exitCode=0 Aug 04 20:12:55.581304 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.581284 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" Aug 04 20:12:55.581499 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.581317 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" event={"ID":"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46","Type":"ContainerDied","Data":"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3"} Aug 04 20:12:55.581499 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.581362 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl" event={"ID":"cc2c9d0b-f3a0-4d9f-8f86-00915674ab46","Type":"ContainerDied","Data":"8011c3de4acbeb3457c31be2d10258521a6289777eb220c115d54bc2230d0211"} Aug 04 20:12:55.581499 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.581384 2567 scope.go:117] "RemoveContainer" containerID="3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3" Aug 04 20:12:55.590478 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.590462 2567 scope.go:117] "RemoveContainer" containerID="3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3" Aug 04 20:12:55.590713 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:12:55.590693 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3\": container with ID starting with 3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3 not found: ID does not exist" containerID="3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3" Aug 04 20:12:55.590780 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.590734 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3"} err="failed to get container status \"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3\": rpc error: code = NotFound desc = could not find container \"3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3\": container with ID starting with 3e39518d7a8877a7e25dd6b26517dce4f87d51fb2d0db38cf7aedb09c43e87f3 not found: ID does not exist" Aug 04 20:12:55.598909 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.598888 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:55.605751 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:55.605733 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f4e87-7b48ccbddd-zxxfl"] Aug 04 20:12:57.505685 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:57.505655 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" path="/var/lib/kubelet/pods/cc2c9d0b-f3a0-4d9f-8f86-00915674ab46/volumes" Aug 04 20:12:58.289790 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:12:58.289760 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:13:01.674653 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:01.674618 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:13:01.675034 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:01.674875 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" containerID="cri-o://ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967" gracePeriod=30 Aug 04 20:13:06.051117 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:06.051081 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 20:13:06.051501 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:06.051152 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 20:13:06.934830 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:06.934802 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:13:07.050440 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050418 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050584 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050455 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050584 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050472 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050584 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050510 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050584 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050537 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050584 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050565 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050839 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050617 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050839 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050676 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050839 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050714 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:07.050839 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f\") pod \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\" (UID: \"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff\") " Aug 04 20:13:07.050839 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050807 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:13:07.051139 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.050894 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:07.051511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.051140 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:07.051511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.051162 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-data\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.051511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.051169 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:07.051511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.051187 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istiod-ca-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.051511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.051204 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.052820 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.052796 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 20:13:07.052820 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.052813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:07.052957 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.052843 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f" (OuterVolumeSpecName: "kube-api-access-zfr4f") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "kube-api-access-zfr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:07.052957 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.052885 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" (UID: "ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:07.152639 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152613 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-token\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.152639 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152636 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-workload-certs\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.152794 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152644 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-kube-api-access-zfr4f\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.152794 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152654 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-podinfo\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.152794 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152662 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-istio-envoy\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.152794 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.152671 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff-credential-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:07.660300 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.660263 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerID="ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967" exitCode=0 Aug 04 20:13:07.660488 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.660318 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" Aug 04 20:13:07.660488 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.660347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerDied","Data":"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967"} Aug 04 20:13:07.660488 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.660404 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826" event={"ID":"ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff","Type":"ContainerDied","Data":"1ec2efd44acfb72dc70cf2405da3e8fa449f14343694950fd7caeeafd352d38f"} Aug 04 20:13:07.660488 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.660442 2567 scope.go:117] "RemoveContainer" containerID="ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967" Aug 04 20:13:07.670006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.669990 2567 scope.go:117] "RemoveContainer" containerID="109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344" Aug 04 20:13:07.679182 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.679155 2567 scope.go:117] "RemoveContainer" containerID="ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967" Aug 04 20:13:07.679447 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:13:07.679422 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967\": container with ID starting with ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967 not found: ID does not exist" containerID="ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967" Aug 04 20:13:07.679500 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.679458 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967"} err="failed to get container status \"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967\": rpc error: code = NotFound desc = could not find container \"ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967\": container with ID starting with ed974d937191fb1eea78d3bab67e1789699504172442bcda3e6e7c382afe0967 not found: ID does not exist" Aug 04 20:13:07.679500 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.679474 2567 scope.go:117] "RemoveContainer" containerID="109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344" Aug 04 20:13:07.679735 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:13:07.679712 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344\": container with ID starting with 109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344 not found: ID does not exist" containerID="109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344" Aug 04 20:13:07.679821 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.679739 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344"} err="failed to get container status \"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344\": rpc error: code = NotFound desc = could not find container \"109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344\": container with ID starting with 109fa7b6539d83a6ee709749aa43184fd0dd7d99fb003a575baa66009e9f8344 not found: ID does not exist" Aug 04 20:13:07.680586 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.680566 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:13:07.684741 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:07.684721 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f4e87-openshift-default-bb6ff5b96-ck826"] Aug 04 20:13:09.505360 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:09.505331 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" path="/var/lib/kubelet/pods/ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff/volumes" Aug 04 20:13:18.699750 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:18.699717 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:13:20.736952 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:20.736916 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:13:20.737435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:20.737173 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" podUID="2359e5b1-5895-486e-af5b-d9b482d50d18" containerName="maas-api" containerID="cri-o://513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b" gracePeriod=30 Aug 04 20:13:21.011707 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.011682 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:13:21.070307 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.070278 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcmp2\" (UniqueName: \"kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2\") pod \"2359e5b1-5895-486e-af5b-d9b482d50d18\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " Aug 04 20:13:21.070470 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.070360 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") pod \"2359e5b1-5895-486e-af5b-d9b482d50d18\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " Aug 04 20:13:21.070470 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.070396 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca\") pod \"2359e5b1-5895-486e-af5b-d9b482d50d18\" (UID: \"2359e5b1-5895-486e-af5b-d9b482d50d18\") " Aug 04 20:13:21.070787 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.070759 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2359e5b1-5895-486e-af5b-d9b482d50d18" (UID: "2359e5b1-5895-486e-af5b-d9b482d50d18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:13:21.072451 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.072427 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2" (OuterVolumeSpecName: "kube-api-access-lcmp2") pod "2359e5b1-5895-486e-af5b-d9b482d50d18" (UID: "2359e5b1-5895-486e-af5b-d9b482d50d18"). InnerVolumeSpecName "kube-api-access-lcmp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:21.072525 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.072511 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2359e5b1-5895-486e-af5b-d9b482d50d18" (UID: "2359e5b1-5895-486e-af5b-d9b482d50d18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:13:21.171006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.170970 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcmp2\" (UniqueName: \"kubernetes.io/projected/2359e5b1-5895-486e-af5b-d9b482d50d18-kube-api-access-lcmp2\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.171006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.171001 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2359e5b1-5895-486e-af5b-d9b482d50d18-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.171006 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.171010 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2359e5b1-5895-486e-af5b-d9b482d50d18-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:13:21.750604 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.750565 2567 generic.go:356] "Generic (PLEG): container finished" podID="2359e5b1-5895-486e-af5b-d9b482d50d18" containerID="513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b" exitCode=0 Aug 04 20:13:21.750604 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.750607 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" Aug 04 20:13:21.751091 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.750623 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" event={"ID":"2359e5b1-5895-486e-af5b-d9b482d50d18","Type":"ContainerDied","Data":"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b"} Aug 04 20:13:21.751091 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.750675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql" event={"ID":"2359e5b1-5895-486e-af5b-d9b482d50d18","Type":"ContainerDied","Data":"646ba7fc87954be0dc1de0a9f1f37f34bd06c053e2785ec31ee3dcd23ecb95ec"} Aug 04 20:13:21.751091 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.750697 2567 scope.go:117] "RemoveContainer" containerID="513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b" Aug 04 20:13:21.759827 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.759803 2567 scope.go:117] "RemoveContainer" containerID="513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b" Aug 04 20:13:21.760115 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:13:21.760096 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b\": container with ID starting with 513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b not found: ID does not exist" containerID="513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b" Aug 04 20:13:21.760189 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.760122 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b"} err="failed to get container status \"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b\": rpc error: code = NotFound desc = could not find container \"513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b\": container with ID starting with 513558fcd6a828af95f59c0cf1008f93dea1bac34c0d23bef388ce5c7035a95b not found: ID does not exist" Aug 04 20:13:21.767737 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.767688 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:13:21.771003 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:21.770984 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41ab60-cdc57d855-srnql"] Aug 04 20:13:23.504916 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:23.504883 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2359e5b1-5895-486e-af5b-d9b482d50d18" path="/var/lib/kubelet/pods/2359e5b1-5895-486e-af5b-d9b482d50d18/volumes" Aug 04 20:13:27.701811 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:13:27.701777 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:14:09.182682 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:09.182647 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:14:09.183191 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:09.182886 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" containerID="cri-o://b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29" gracePeriod=30 Aug 04 20:14:14.432458 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.432434 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:14:14.552714 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552691 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.552854 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552719 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.552854 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.552854 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jll\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.552854 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552824 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.552854 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552845 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.553146 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.553146 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552942 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.553146 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.552979 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs\") pod \"4b26fe7d-c0af-4282-85ce-36002354ef44\" (UID: \"4b26fe7d-c0af-4282-85ce-36002354ef44\") " Aug 04 20:14:14.553303 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553174 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:14.553303 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553277 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:14.553425 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553397 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-credential-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.553485 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553430 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b26fe7d-c0af-4282-85ce-36002354ef44-istiod-ca-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.553485 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553389 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:14.553583 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553563 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:14.553627 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.553559 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:14.555005 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.554977 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll" (OuterVolumeSpecName: "kube-api-access-97jll") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "kube-api-access-97jll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:14.555143 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.555006 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:14.555143 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.555037 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 20:14:14.555267 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.555185 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b26fe7d-c0af-4282-85ce-36002354ef44" (UID: "4b26fe7d-c0af-4282-85ce-36002354ef44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:14.654262 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654238 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-token\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654262 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654258 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654262 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654267 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-data\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654276 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-workload-certs\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654283 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-envoy\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654290 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b26fe7d-c0af-4282-85ce-36002354ef44-istio-podinfo\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:14.654447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:14.654298 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jll\" (UniqueName: \"kubernetes.io/projected/4b26fe7d-c0af-4282-85ce-36002354ef44-kube-api-access-97jll\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:15.051990 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.051954 2567 generic.go:356] "Generic (PLEG): container finished" podID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerID="b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29" exitCode=0 Aug 04 20:14:15.052204 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.052013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerDied","Data":"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29"} Aug 04 20:14:15.052204 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.052040 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" Aug 04 20:14:15.052204 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.052105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt" event={"ID":"4b26fe7d-c0af-4282-85ce-36002354ef44","Type":"ContainerDied","Data":"0af0b423b0fad948f1f657d6fbbf592ec22bcde0c25ecfe255c7377a1dd1acec"} Aug 04 20:14:15.052204 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.052128 2567 scope.go:117] "RemoveContainer" containerID="b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29" Aug 04 20:14:15.062918 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.062897 2567 scope.go:117] "RemoveContainer" containerID="10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605" Aug 04 20:14:15.072558 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.072540 2567 scope.go:117] "RemoveContainer" containerID="b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29" Aug 04 20:14:15.072795 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:15.072778 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29\": container with ID starting with b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29 not found: ID does not exist" containerID="b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29" Aug 04 20:14:15.072844 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.072801 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29"} err="failed to get container status \"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29\": rpc error: code = NotFound desc = could not find container \"b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29\": container with ID starting with b1360b91b92ec33e8f731f4d7975d64392deac948de9621a3c0c9f80a98bab29 not found: ID does not exist" Aug 04 20:14:15.072844 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.072817 2567 scope.go:117] "RemoveContainer" containerID="10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605" Aug 04 20:14:15.073008 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:15.072991 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605\": container with ID starting with 10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605 not found: ID does not exist" containerID="10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605" Aug 04 20:14:15.073078 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.073012 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605"} err="failed to get container status \"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605\": rpc error: code = NotFound desc = could not find container \"10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605\": container with ID starting with 10854b4c965178b340d48a47c3fdec0f13de50d2570d309079af0d6d31c69605 not found: ID does not exist" Aug 04 20:14:15.079579 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.079558 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:14:15.085945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.085925 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-41ab60-openshift-default-8457ccf594-g9frt"] Aug 04 20:14:15.504816 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:15.504735 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" path="/var/lib/kubelet/pods/4b26fe7d-c0af-4282-85ce-36002354ef44/volumes" Aug 04 20:14:23.189382 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:23.189348 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:14:27.533705 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:27.533673 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:14:27.534187 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:27.533896 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" containerID="cri-o://14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e" gracePeriod=30 Aug 04 20:14:27.808258 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:27.808179 2567 patch_prober.go:28] interesting pod/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 20:14:27.808258 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:27.808241 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 20:14:32.775139 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.775117 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:14:32.913869 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.913794 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914013 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.913866 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxbxh\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914013 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.913912 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914013 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.913939 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914013 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.913965 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914282 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914040 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914282 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914091 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:32.914282 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914128 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914282 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914169 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914282 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914243 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo\") pod \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\" (UID: \"babfdfc6-3c4c-47cd-aa72-0308618be1f6\") " Aug 04 20:14:32.914543 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914264 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:32.914543 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914336 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:32.914543 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914482 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:32.914697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914576 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-data\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:32.914697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914595 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-certs\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:32.914697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:32.914697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914609 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-credential-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:32.914697 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.914626 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-workload-socket\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:32.916145 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.916119 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:32.916394 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.916377 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh" (OuterVolumeSpecName: "kube-api-access-qxbxh") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "kube-api-access-qxbxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:32.916614 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.916593 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:32.916614 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:32.916604 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "babfdfc6-3c4c-47cd-aa72-0308618be1f6" (UID: "babfdfc6-3c4c-47cd-aa72-0308618be1f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 20:14:33.015358 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.015336 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-podinfo\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:33.015358 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.015356 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbxh\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-kube-api-access-qxbxh\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:33.015479 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.015366 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-token\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:33.015479 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.015374 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istio-envoy\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:33.015479 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.015382 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babfdfc6-3c4c-47cd-aa72-0308618be1f6-istiod-ca-cert\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:33.156609 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.156581 2567 generic.go:356] "Generic (PLEG): container finished" podID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerID="14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e" exitCode=0 Aug 04 20:14:33.156740 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.156628 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" Aug 04 20:14:33.156740 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.156672 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerDied","Data":"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e"} Aug 04 20:14:33.156838 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.156750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk" event={"ID":"babfdfc6-3c4c-47cd-aa72-0308618be1f6","Type":"ContainerDied","Data":"f684e9c4d353d47ecc5993bfada4ff987cc6f729ef520db7107d89acd85620cc"} Aug 04 20:14:33.156838 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.156774 2567 scope.go:117] "RemoveContainer" containerID="14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e" Aug 04 20:14:33.165928 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.165914 2567 scope.go:117] "RemoveContainer" containerID="22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e" Aug 04 20:14:33.173688 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.173673 2567 scope.go:117] "RemoveContainer" containerID="14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e" Aug 04 20:14:33.173913 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:33.173897 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e\": container with ID starting with 14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e not found: ID does not exist" containerID="14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e" Aug 04 20:14:33.173961 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.173921 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e"} err="failed to get container status \"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e\": rpc error: code = NotFound desc = could not find container \"14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e\": container with ID starting with 14380f23e5e868f990c40a1a8c38ab329f10847cc047a749f33ba4173ac06c9e not found: ID does not exist" Aug 04 20:14:33.173961 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.173936 2567 scope.go:117] "RemoveContainer" containerID="22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e" Aug 04 20:14:33.174183 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:33.174168 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e\": container with ID starting with 22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e not found: ID does not exist" containerID="22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e" Aug 04 20:14:33.174237 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.174187 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e"} err="failed to get container status \"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e\": rpc error: code = NotFound desc = could not find container \"22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e\": container with ID starting with 22d3dbf5b71db56af5118250bf4a3d6a9b89facfade24568f070445c35cb1a7e not found: ID does not exist" Aug 04 20:14:33.179144 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.179125 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:14:33.183209 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.183191 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9cd851-openshift-default-847cccbbd-97smk"] Aug 04 20:14:33.504467 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:33.504390 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" path="/var/lib/kubelet/pods/babfdfc6-3c4c-47cd-aa72-0308618be1f6/volumes" Aug 04 20:14:38.453041 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.453009 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 20:14:38.453537 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.453273 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" podUID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" containerName="maas-api" containerID="cri-o://68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851" gracePeriod=30 Aug 04 20:14:38.483035 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.483012 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b"] Aug 04 20:14:38.488239 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.488218 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-03ae76-29764560-zdg7b"] Aug 04 20:14:38.702468 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.702447 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 20:14:38.857322 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.857298 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") pod \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " Aug 04 20:14:38.857477 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.857349 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9knbx\" (UniqueName: \"kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx\") pod \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " Aug 04 20:14:38.857477 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.857375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca\") pod \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\" (UID: \"987ede6f-2ce2-4138-b32d-b6f9ec917cfb\") " Aug 04 20:14:38.857746 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.857725 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "987ede6f-2ce2-4138-b32d-b6f9ec917cfb" (UID: "987ede6f-2ce2-4138-b32d-b6f9ec917cfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:38.859448 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.859423 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "987ede6f-2ce2-4138-b32d-b6f9ec917cfb" (UID: "987ede6f-2ce2-4138-b32d-b6f9ec917cfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:14:38.859566 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.859479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx" (OuterVolumeSpecName: "kube-api-access-9knbx") pod "987ede6f-2ce2-4138-b32d-b6f9ec917cfb" (UID: "987ede6f-2ce2-4138-b32d-b6f9ec917cfb"). InnerVolumeSpecName "kube-api-access-9knbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:38.958166 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.958138 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:38.958166 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.958160 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9knbx\" (UniqueName: \"kubernetes.io/projected/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-kube-api-access-9knbx\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:38.958166 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:38.958170 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ede6f-2ce2-4138-b32d-b6f9ec917cfb-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:39.193297 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.193214 2567 generic.go:356] "Generic (PLEG): container finished" podID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" containerID="68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851" exitCode=0 Aug 04 20:14:39.193297 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.193273 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" Aug 04 20:14:39.193512 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.193292 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" event={"ID":"987ede6f-2ce2-4138-b32d-b6f9ec917cfb","Type":"ContainerDied","Data":"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851"} Aug 04 20:14:39.193512 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.193364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b" event={"ID":"987ede6f-2ce2-4138-b32d-b6f9ec917cfb","Type":"ContainerDied","Data":"1e677f55b12e7e353d9c228f1fef969da46caa1fe2899fbe182fbbfb4b3f26c9"} Aug 04 20:14:39.193512 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.193387 2567 scope.go:117] "RemoveContainer" containerID="68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851" Aug 04 20:14:39.202183 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.202133 2567 scope.go:117] "RemoveContainer" containerID="68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851" Aug 04 20:14:39.202414 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:39.202394 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851\": container with ID starting with 68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851 not found: ID does not exist" containerID="68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851" Aug 04 20:14:39.202474 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.202425 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851"} err="failed to get container status \"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851\": rpc error: code = NotFound desc = could not find container \"68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851\": container with ID starting with 68b0b688ba565fdfcef40c2186c3d2e390390a396dcbd783953c46f4eaea1851 not found: ID does not exist" Aug 04 20:14:39.214557 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.214536 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 20:14:39.219705 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.219685 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-03ae76-579dcbb56c-mjb4b"] Aug 04 20:14:39.505988 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.505908 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" path="/var/lib/kubelet/pods/987ede6f-2ce2-4138-b32d-b6f9ec917cfb/volumes" Aug 04 20:14:39.506429 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:39.506315 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ada90a-f489-466e-b605-7d684c30f74b" path="/var/lib/kubelet/pods/d6ada90a-f489-466e-b605-7d684c30f74b/volumes" Aug 04 20:14:40.051713 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:40.051688 2567 scope.go:117] "RemoveContainer" containerID="0519b969e1b61e049017be0c45450953c0c4a2f5e98bb6e1c53139b4308c8c0a" Aug 04 20:14:40.787739 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:40.787709 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:14:49.618620 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:49.618585 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 20:14:49.619229 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:49.618934 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" podUID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" containerName="maas-api" containerID="cri-o://271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf" gracePeriod=30 Aug 04 20:14:49.865215 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:49.865191 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 20:14:50.049106 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.049082 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt2qh\" (UniqueName: \"kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh\") pod \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " Aug 04 20:14:50.049253 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.049193 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca\") pod \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " Aug 04 20:14:50.049253 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.049222 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") pod \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\" (UID: \"f0547b7e-fbd7-4e8f-8087-668c857ad71e\") " Aug 04 20:14:50.049551 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.049527 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0547b7e-fbd7-4e8f-8087-668c857ad71e" (UID: "f0547b7e-fbd7-4e8f-8087-668c857ad71e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:50.051021 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.051000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh" (OuterVolumeSpecName: "kube-api-access-rt2qh") pod "f0547b7e-fbd7-4e8f-8087-668c857ad71e" (UID: "f0547b7e-fbd7-4e8f-8087-668c857ad71e"). InnerVolumeSpecName "kube-api-access-rt2qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:50.051110 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.051084 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0547b7e-fbd7-4e8f-8087-668c857ad71e" (UID: "f0547b7e-fbd7-4e8f-8087-668c857ad71e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:14:50.150385 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.150346 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547b7e-fbd7-4e8f-8087-668c857ad71e-openshift-service-ca\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.150385 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.150385 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547b7e-fbd7-4e8f-8087-668c857ad71e-maas-api-tls\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.150566 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.150399 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt2qh\" (UniqueName: \"kubernetes.io/projected/f0547b7e-fbd7-4e8f-8087-668c857ad71e-kube-api-access-rt2qh\") on node \"ip-10-0-140-250.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.253861 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.253827 2567 generic.go:356] "Generic (PLEG): container finished" podID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" containerID="271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf" exitCode=0 Aug 04 20:14:50.253987 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.253872 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" Aug 04 20:14:50.253987 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.253893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" event={"ID":"f0547b7e-fbd7-4e8f-8087-668c857ad71e","Type":"ContainerDied","Data":"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf"} Aug 04 20:14:50.253987 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.253944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98" event={"ID":"f0547b7e-fbd7-4e8f-8087-668c857ad71e","Type":"ContainerDied","Data":"410e7cdf3d9c1fcf004c9f800d50ba5d2ae352f38c61f2ffa2106ec9725ad023"} Aug 04 20:14:50.253987 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.253975 2567 scope.go:117] "RemoveContainer" containerID="271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf" Aug 04 20:14:50.263099 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.263081 2567 scope.go:117] "RemoveContainer" containerID="271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf" Aug 04 20:14:50.263331 ip-10-0-140-250 kubenswrapper[2567]: E0804 20:14:50.263311 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf\": container with ID starting with 271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf not found: ID does not exist" containerID="271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf" Aug 04 20:14:50.263377 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.263339 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf"} err="failed to get container status \"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf\": rpc error: code = NotFound desc = could not find container \"271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf\": container with ID starting with 271f800e0fea0c7207fbdb34704c99da3eb0b1397068a47b308e51cea12c1ebf not found: ID does not exist" Aug 04 20:14:50.275848 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.275828 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 20:14:50.281627 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:50.281604 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-15a874-7bfb9f66c8-gts98"] Aug 04 20:14:51.504355 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:51.504323 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" path="/var/lib/kubelet/pods/f0547b7e-fbd7-4e8f-8087-668c857ad71e/volumes" Aug 04 20:14:51.797178 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:14:51.797141 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-njkqg"] Aug 04 20:15:01.942635 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:01.942584 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nr7x5_8980ee69-e170-494f-a6d8-87601698a3e2/manager/0.log" Aug 04 20:15:02.060606 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:02.060573 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6zqfz_6f9a58f8-acec-42f4-88c0-0d5a4c93e488/manager/0.log" Aug 04 20:15:02.418837 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:02.418810 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b452j_ebe59b4a-e165-4a39-a277-b74d18a9e17d/manager/0.log" Aug 04 20:15:02.640877 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:02.640852 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-k7h2c_1d52f445-4f93-4c04-8de7-e44602ed39fe/copy-manifests/0.log" Aug 04 20:15:02.647436 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:02.647411 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-k7h2c_1d52f445-4f93-4c04-8de7-e44602ed39fe/manager/0.log" Aug 04 20:15:03.241545 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:03.241516 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764530-4svsq_e79879ad-0384-49be-9171-b578704b9c53/cleanup/0.log" Aug 04 20:15:03.349961 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:03.349934 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764545-9fkwk_daee190c-6c13-4229-9728-02a2aa0bf325/cleanup/0.log" Aug 04 20:15:03.459100 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:03.459075 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764560-t9pgs_3980cde5-0a59-4cd4-84bf-db590f40d427/cleanup/0.log" Aug 04 20:15:03.617644 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:03.617613 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq"] Aug 04 20:15:03.619700 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:03.619682 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764530-4svsq"] Aug 04 20:15:04.531999 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:04.531972 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-bpj2c_37cca5ab-b6db-4c04-9f43-b91d328aa13b/manager/0.log" Aug 04 20:15:04.979173 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:04.979143 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-njkqg_64d3ba82-bbdf-408a-a47c-8a730eb58b4f/limitador/0.log" Aug 04 20:15:05.097560 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:05.097526 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-xzkpg_f75d7b5a-0ca4-4064-b208-7927b09d0cac/manager/0.log" Aug 04 20:15:05.504340 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:05.504310 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79879ad-0384-49be-9171-b578704b9c53" path="/var/lib/kubelet/pods/e79879ad-0384-49be-9171-b578704b9c53/volumes" Aug 04 20:15:05.556217 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:05.556190 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrhcn_ea0cde63-27c5-4218-90f9-777a375b8855/discovery/0.log" Aug 04 20:15:05.664511 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:05.664488 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-bqdrm_c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29/kube-auth-proxy/0.log" Aug 04 20:15:12.834795 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.834756 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bk9n8/must-gather-smkhj"] Aug 04 20:15:12.835346 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835308 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.835346 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835328 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.835346 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835342 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835351 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835373 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835382 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835392 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835401 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835410 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835419 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835439 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835449 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835461 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835470 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835481 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835491 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835502 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835510 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835530 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2359e5b1-5895-486e-af5b-d9b482d50d18" containerName="maas-api" Aug 04 20:15:12.835575 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835539 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2359e5b1-5895-486e-af5b-d9b482d50d18" containerName="maas-api" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835624 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0547b7e-fbd7-4e8f-8087-668c857ad71e" containerName="maas-api" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835637 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2c9d0b-f3a0-4d9f-8f86-00915674ab46" containerName="maas-api" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835648 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835659 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835668 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2359e5b1-5895-486e-af5b-d9b482d50d18" containerName="maas-api" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835679 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835688 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b26fe7d-c0af-4282-85ce-36002354ef44" containerName="istio-proxy" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835697 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="987ede6f-2ce2-4138-b32d-b6f9ec917cfb" containerName="maas-api" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835874 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1cb84a-0e8b-4aae-9dbe-4f2e0357bfff" containerName="istio-proxy" Aug 04 20:15:12.836435 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.835891 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="babfdfc6-3c4c-47cd-aa72-0308618be1f6" containerName="istio-proxy" Aug 04 20:15:12.839402 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.839381 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:12.841751 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.841722 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"openshift-service-ca.crt\"" Aug 04 20:15:12.841869 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.841807 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"kube-root-ca.crt\"" Aug 04 20:15:12.849618 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.849600 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/must-gather-smkhj"] Aug 04 20:15:12.922698 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.922671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7vh\" (UniqueName: \"kubernetes.io/projected/ee93470b-13fe-45c2-a04f-25ed4f9064bf-kube-api-access-9k7vh\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:12.922857 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:12.922712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee93470b-13fe-45c2-a04f-25ed4f9064bf-must-gather-output\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.023249 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.023223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7vh\" (UniqueName: \"kubernetes.io/projected/ee93470b-13fe-45c2-a04f-25ed4f9064bf-kube-api-access-9k7vh\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.023382 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.023260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee93470b-13fe-45c2-a04f-25ed4f9064bf-must-gather-output\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.023662 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.023644 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee93470b-13fe-45c2-a04f-25ed4f9064bf-must-gather-output\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.037995 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.037970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7vh\" (UniqueName: \"kubernetes.io/projected/ee93470b-13fe-45c2-a04f-25ed4f9064bf-kube-api-access-9k7vh\") pod \"must-gather-smkhj\" (UID: \"ee93470b-13fe-45c2-a04f-25ed4f9064bf\") " pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.149475 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.149407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/must-gather-smkhj" Aug 04 20:15:13.275515 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.275486 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/must-gather-smkhj"] Aug 04 20:15:13.276992 ip-10-0-140-250 kubenswrapper[2567]: W0804 20:15:13.276968 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee93470b_13fe_45c2_a04f_25ed4f9064bf.slice/crio-9abfa9fadc4ab893f070f74f3a72815d306fdac390102ef35166e75af171cfe5 WatchSource:0}: Error finding container 9abfa9fadc4ab893f070f74f3a72815d306fdac390102ef35166e75af171cfe5: Status 404 returned error can't find the container with id 9abfa9fadc4ab893f070f74f3a72815d306fdac390102ef35166e75af171cfe5 Aug 04 20:15:13.278802 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.278777 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 20:15:13.386699 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:13.386668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/must-gather-smkhj" event={"ID":"ee93470b-13fe-45c2-a04f-25ed4f9064bf","Type":"ContainerStarted","Data":"9abfa9fadc4ab893f070f74f3a72815d306fdac390102ef35166e75af171cfe5"} Aug 04 20:15:14.395012 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:14.394982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/must-gather-smkhj" event={"ID":"ee93470b-13fe-45c2-a04f-25ed4f9064bf","Type":"ContainerStarted","Data":"e6736cae6c47c1cee65ea321248979c787bbd879b69e230daef0dd4310eab027"} Aug 04 20:15:15.403998 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:15.403962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/must-gather-smkhj" event={"ID":"ee93470b-13fe-45c2-a04f-25ed4f9064bf","Type":"ContainerStarted","Data":"4c1d6e03a5c9dff67d520d91b6d0c77c45df5f4d1f49ff8e06b1f700ef4d1eb2"} Aug 04 20:15:15.419788 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:15.419735 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bk9n8/must-gather-smkhj" podStartSLOduration=2.462163121 podStartE2EDuration="3.419719616s" totalImagesPullingTime="957.556495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:15:15.418365183 +0000 UTC m=+3576.388391702" watchObservedRunningTime="2026-08-04 20:15:15.419719616 +0000 UTC m=+3576.389746135" Aug 04 20:15:15.734747 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:15.734677 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b4hwg_524abca7-3ea5-44a3-a90e-fa2c5b9e6c96/global-pull-secret-syncer/0.log" Aug 04 20:15:15.914945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:15.914913 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7tlx9_6c51e61e-eef9-4ea5-8e9c-f7a8eb65a338/konnectivity-agent/0.log" Aug 04 20:15:16.009802 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:16.009715 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-250.ec2.internal_ffeedbcd8e5be3b25f752d90133cc1ac/haproxy/0.log" Aug 04 20:15:20.967443 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:20.967406 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-bpj2c_37cca5ab-b6db-4c04-9f43-b91d328aa13b/manager/0.log" Aug 04 20:15:21.212569 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:21.212529 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-njkqg_64d3ba82-bbdf-408a-a47c-8a730eb58b4f/limitador/0.log" Aug 04 20:15:21.337256 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:21.337223 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-xzkpg_f75d7b5a-0ca4-4064-b208-7927b09d0cac/manager/0.log" Aug 04 20:15:22.668628 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.668595 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/alertmanager/0.log" Aug 04 20:15:22.698064 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.698007 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/config-reloader/0.log" Aug 04 20:15:22.719736 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.719700 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/kube-rbac-proxy-web/0.log" Aug 04 20:15:22.744108 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.744084 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/kube-rbac-proxy/0.log" Aug 04 20:15:22.785945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.785921 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/kube-rbac-proxy-metric/0.log" Aug 04 20:15:22.815729 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.815706 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/prom-label-proxy/0.log" Aug 04 20:15:22.839542 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.839512 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_78f583ca-7bff-40a7-b036-eae5039b7e6b/init-config-reloader/0.log" Aug 04 20:15:22.932029 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.931919 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-q4gdn_a5a17e53-02ff-463a-9919-160910ac2e10/kube-state-metrics/0.log" Aug 04 20:15:22.956386 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.956358 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-q4gdn_a5a17e53-02ff-463a-9919-160910ac2e10/kube-rbac-proxy-main/0.log" Aug 04 20:15:22.998083 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:22.991087 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-q4gdn_a5a17e53-02ff-463a-9919-160910ac2e10/kube-rbac-proxy-self/0.log" Aug 04 20:15:23.041786 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.041718 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5554d79c5f-k4j8w_aa2e0347-ff51-4827-8107-d979e9b9c673/metrics-server/0.log" Aug 04 20:15:23.281945 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.281868 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkppb_08a85812-f966-4d91-bfe4-39a86252ea9a/node-exporter/0.log" Aug 04 20:15:23.330847 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.330814 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkppb_08a85812-f966-4d91-bfe4-39a86252ea9a/kube-rbac-proxy/0.log" Aug 04 20:15:23.355851 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.355821 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkppb_08a85812-f966-4d91-bfe4-39a86252ea9a/init-textfile/0.log" Aug 04 20:15:23.819315 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.819268 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9f54f686-rp52c_f384657b-1743-4c2e-b3fc-635dad005b56/telemeter-client/0.log" Aug 04 20:15:23.848412 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.848382 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9f54f686-rp52c_f384657b-1743-4c2e-b3fc-635dad005b56/reload/0.log" Aug 04 20:15:23.864221 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:23.864196 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9f54f686-rp52c_f384657b-1743-4c2e-b3fc-635dad005b56/kube-rbac-proxy/0.log" Aug 04 20:15:25.622683 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.622646 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt"] Aug 04 20:15:25.630768 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.630737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.636605 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.636577 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bk9n8\"/\"default-dockercfg-qgnpn\"" Aug 04 20:15:25.638557 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.638531 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt"] Aug 04 20:15:25.754771 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.754740 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-sys\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.754952 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.754804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-lib-modules\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.754952 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.754881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-podres\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.754952 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.754921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-proc\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.755084 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.754963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfz9\" (UniqueName: \"kubernetes.io/projected/27cc838c-6094-451f-985a-f734d0ccf7d8-kube-api-access-wkfz9\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.856268 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.856231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-lib-modules\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.856717 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.856697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-podres\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.856976 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.856961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-proc\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.857209 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.857192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfz9\" (UniqueName: \"kubernetes.io/projected/27cc838c-6094-451f-985a-f734d0ccf7d8-kube-api-access-wkfz9\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.857747 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.857727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-sys\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.857891 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.857139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-proc\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.857891 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.856913 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-podres\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.858010 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.856619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-lib-modules\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.858010 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.857789 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27cc838c-6094-451f-985a-f734d0ccf7d8-sys\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.866274 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.866248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfz9\" (UniqueName: \"kubernetes.io/projected/27cc838c-6094-451f-985a-f734d0ccf7d8-kube-api-access-wkfz9\") pod \"perf-node-gather-daemonset-rpgwt\" (UID: \"27cc838c-6094-451f-985a-f734d0ccf7d8\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:25.944695 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:25.944614 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:26.118890 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.118867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt"] Aug 04 20:15:26.326620 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.326591 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-695c89c559-kmlj2_fe0b59e5-e704-43f1-97a2-13b888e71392/console/0.log" Aug 04 20:15:26.498647 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.498559 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" event={"ID":"27cc838c-6094-451f-985a-f734d0ccf7d8","Type":"ContainerStarted","Data":"a5fdb78b63d1e5a102d3d2d836ef0f99c6701f1edd10e33aafa14c5d14409d47"} Aug 04 20:15:26.498647 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.498614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" event={"ID":"27cc838c-6094-451f-985a-f734d0ccf7d8","Type":"ContainerStarted","Data":"ecf69de0bb145203ce1e422c7fafb49c7852aa0a6cf35aba53db0eabd98347a3"} Aug 04 20:15:26.498876 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.498769 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:26.518686 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:26.518629 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" podStartSLOduration=1.5186087860000002 podStartE2EDuration="1.518608786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:15:26.51544363 +0000 UTC m=+3587.485470147" watchObservedRunningTime="2026-08-04 20:15:26.518608786 +0000 UTC m=+3587.488635307" Aug 04 20:15:28.006690 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:28.006644 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c8mm_3db70ed7-f73a-47ca-a650-a124d82fb08c/dns/0.log" Aug 04 20:15:28.026684 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:28.026655 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c8mm_3db70ed7-f73a-47ca-a650-a124d82fb08c/kube-rbac-proxy/0.log" Aug 04 20:15:28.193350 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:28.193315 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwb2t_c11e729b-8b85-4b6a-a86f-3a58fb19471e/dns-node-resolver/0.log" Aug 04 20:15:28.708138 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:28.708106 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-ffd8f474-bcmbw_7b55cc7a-4785-4725-900f-97ac50a1658e/registry/0.log" Aug 04 20:15:28.778150 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:28.778118 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k46fv_0b5c4ce2-5922-41bb-8802-f0b554e441b4/node-ca/0.log" Aug 04 20:15:30.023717 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:30.023687 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrhcn_ea0cde63-27c5-4218-90f9-777a375b8855/discovery/0.log" Aug 04 20:15:30.044149 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:30.044113 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-bqdrm_c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29/kube-auth-proxy/0.log" Aug 04 20:15:31.014427 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:31.014396 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fcklt_cc5463a9-37d5-45ae-84f1-dc3858b8c6c9/serve-healthcheck-canary/0.log" Aug 04 20:15:31.514159 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:31.514123 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pvgm_bc129f0a-75ee-4332-a393-60520f6c4ac7/kube-rbac-proxy/0.log" Aug 04 20:15:31.534396 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:31.534370 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pvgm_bc129f0a-75ee-4332-a393-60520f6c4ac7/exporter/0.log" Aug 04 20:15:31.555449 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:31.555424 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pvgm_bc129f0a-75ee-4332-a393-60520f6c4ac7/extractor/0.log" Aug 04 20:15:32.521163 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:32.521133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-rpgwt" Aug 04 20:15:33.985974 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:33.985947 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nr7x5_8980ee69-e170-494f-a6d8-87601698a3e2/manager/0.log" Aug 04 20:15:34.436749 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:34.436407 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6zqfz_6f9a58f8-acec-42f4-88c0-0d5a4c93e488/manager/0.log" Aug 04 20:15:34.756662 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:34.756611 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b452j_ebe59b4a-e165-4a39-a277-b74d18a9e17d/manager/0.log" Aug 04 20:15:34.829931 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:34.829854 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-k7h2c_1d52f445-4f93-4c04-8de7-e44602ed39fe/manager/0.log" Aug 04 20:15:34.849104 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:34.849077 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-k7h2c_1d52f445-4f93-4c04-8de7-e44602ed39fe/copy-manifests/0.log" Aug 04 20:15:36.307447 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:36.307418 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-g6l84_f7b9986e-22fe-4e4f-bb52-98dd4f1a1cc5/openshift-lws-operator/0.log" Aug 04 20:15:36.505863 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:36.505784 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrhcn_ea0cde63-27c5-4218-90f9-777a375b8855/discovery/0.log" Aug 04 20:15:36.515014 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:36.514985 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-bqdrm_c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29/kube-auth-proxy/0.log" Aug 04 20:15:38.263732 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:38.263703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-wrhcn_ea0cde63-27c5-4218-90f9-777a375b8855/discovery/0.log" Aug 04 20:15:38.281980 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:38.281954 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcb8f55f4-bqdrm_c2bae692-36bf-4d6f-8fb2-46a1eb4a2f29/kube-auth-proxy/0.log" Aug 04 20:15:40.083512 ip-10-0-140-250 kubenswrapper[2567]: I0804 20:15:40.083484 2567 scope.go:117] "RemoveContainer" containerID="a4a9ee93e7ae96a46d93e5e11c335211d0e6e47c60ec4d17e63928ab536d0b7d"